Today I’d like to share my experience in iOS video streaming. I’m currently working on my first iOS project, which requires to set up a video stream between nearby iPhones, connected via Wi-Fi or Bluetooth. Devices communication is handled by Multipeer Connectivity Framework (MPC). If you’re not familiar with MPC, check out our previous post Introduction in Multipeer Connectivity Framework. Basic concepts and examples.

Brief overview of the task

Let’s consider the basic scenario. Assume we have two iOS devices, one of which captures video and streams it, while another receives video data and displays it on the screen.

Basically, we’ll need to set up a capture session on a “sender” device, do some image preprocessing, and set up a multipeer connectivity session to connect the devices and stream video data.

Capture session and image processing

For video recording we’ll create an AVCaptureSession object. It allows to configure a media capture process and handles data flow from capture inputs (such as data from camera or microphone) to media outputs (as video and audio data for displaying on screen or saving to file).
Here’s what we’ll need to start.

 import AVFoundation

 var videoDevice: AVCaptureDevice?
 var captureSession: AVCaptureSession!
 var videoDeviceInput : AVCaptureDeviceInput!

 var videoDeviceOutput: AVCaptureVideoDataOutput!
 var videoPreviewLayer: AVCaptureVideoPreviewLayer?
 var videoOutputQueue = DispatchQueue(label: "video_output_queue", attributes: .concurrent)  
 
class VideoViewController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
    override func viewDidLoad() {
        super.viewDidLoad()
        // Here we'll create and configure a video capture session.
        ...
    }
}

Configuring capture session

Let’s set up a capture session to capture video input from the camera and preview it on the screen.

      // First, let's create a capture session.
     captureSession = AVCaptureSession()        
     captureSession.beginConfiguration()  
     // Select a capture device
     let videoDevice = AVCaptureDevice.default(.builtInWideAngleCamera,
                                          for: .video, position: .back)  

Add a video device input to capture video from selected capture device.

      // Add video input to the capture session
        guard
            let videoDeviceInput = try? AVCaptureDeviceInput(device: videoDevice!),
            captureSession.canAddInput(videoDeviceInput)
            else {
                print("Error: Can`t add capture input.")
                return
            }
            print("--- Added video input to capture session. ---")

Add video output to preview the captured data and send it to the receiver.

       // Add video output to the capture session
        videoDeviceOutput = AVCaptureVideoDataOutput()
        videoDeviceOutput.alwaysDiscardsLateVideoFrames = true
        videoDeviceOutput.setSampleBufferDelegate(self, queue: self.videoOutputQueue)      

        videoOutputQueue.async {
         
            if self.captureSession.canAddOutput(self.videoDeviceOutput) {
                self.captureSession.addOutput(self.videoDeviceOutput)
                print("--- Added video output to capture session. ---")
            }
        }
    
        // Set up a session preset
        captureSession.sessionPreset = .medium
        captureSession.commitConfiguration()

Add a video preview layer to display captured video on the screen.

          // Add preview layer for the captured video
        videoPreviewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
        videoPreviewLayer?.videoGravity = AVLayerVideoGravity.resizeAspectFill
        videoPreviewLayer?.frame = view.layer.bounds
        view.layer.addSublayer(videoPreviewLayer!)        

To start/stop the capture session, use the following:

   // Start capture session
    captureSession.startRunning()
    // Stop capture session
    captureSession.stopRunning()

When camera captures new video frame, AVCaptureVideoDataOutput calls delegate method captureOutput(_:didOutput:from:). That’s where we’ll process freshly captured video frames and send it straight to the receiving device.

 func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        // Get image data from sampleBuffer
        guard let imageBuffer: CVImageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
            else {
                NSLog("--- Failed to get an image buffer ---")
                return
        }
        // Convert captured frame data to RGB and prepare for sending
        ...
        // Send converted frame to nearby peer 
        ...
        }

Processing captured video frames

Camera captures video frames in YpCbCr format. To stream captured images to receiver’s image view, we’ll convert it to ARGB format first. Here’s an official guide on how to do this: Converting Luminance and Chrominance Planes to an ARGB Image. I won’t describe the process in detail in here, as this topic deserves another blog post. If you’re looking for a ready-to-go Swift implementation, check out this link.

I used Accelerate framework for more effective image processing:

 import Accelerate

Converted video frame data will be written to vImage_Buffer. Declare it as a class member to avoid reinitializing for each frame.

  var destinationBuffer = vImage_Buffer()

Following steps explained in Apple’s guide, we’ll get captured video data converted to RGB color space and written to destinationBuffer.

        // Convert captured frame data to RGB
        convertYpCbCrToRGB(from: imageBuffer, to: &self.destinationBuffer)
       
        var error = kvImageNoError        
        // Set up image format
        var format = vImage_CGImageFormat(
            bitsPerComponent: 8,
            bitsPerPixel: 32,
            colorSpace: nil,
            bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.first.rawValue),
            version: 0,
            decode: nil,
            renderingIntent: .defaultIntent)       

        // Create cgImage from converted data
        let cgImage = vImageCreateCGImageFromBuffer(
            &destinationBuffer,
            &format,
            nil,
            nil,
            vImage_Flags(kvImageNoFlags),
            &error)

Now our video data is prepared for sending.

Multipeer Connectivity session

Multipeer Connectivity Framework supports discovery of nearby devices, setting up connections, and maintaining communication between connected peers with data exchange (such as sending messages and resources or streaming).
You can find a comprehensive explanation of the framework’s basics and sample code in our previous post.
Don’t forget to

   import  MultipeerConnectivity
 

Here’s a very basic setup.

    // Multipeer session
    var session: MCSession!
    var peer: MCPeerID!
    var browser: MCNearbyServiceBrowser!
    var advertiser: MCNearbyServiceAdvertiser!
    
    let serviceType = "video-streaming"
    peer = MCPeerID(displayName: UIDevice.current.name)        
    // MCSession supports communication between nearby peers
    session = MCSession(peer: peer, securityIdentity: nil, encryptionPreference: MCEncryptionPreference.optional)
    session.delegate = self

    // Browser searches for nearby devices
    browser = MCNearbyServiceBrowser(peer: peer, serviceType: serviceType)
    browser.delegate = self

    // Advertiser provides information for peer to be discovered and invited to the session
    advertiser = MCNearbyServiceAdvertiser(peer: peer, discoveryInfo: nil, serviceType: serviceType)
    advertiser.delegate = self   

Set one of the devices to browse, and another one – to advertise. For details check the MPC-links above.

Sending data

Now when both capture session and mulipeer connectivity session are set up, we can grab video frames received by captureOutput(_:didOutput:from:) method and send them to connected peer.

func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        ...
        // Send converted frame to nearby peer 
        DispatchQueue.main.async {
            if let cgImage = cgImage {
                let image = UIImage(cgImage: cgImage.takeRetainedValue())
                // Compress image data
                let jpegData = UIImageJPEGRepresentation(image, 0.75)
                // Send image data to target peer
                try? session?.send(jpegData!, toPeers: [target!], with: .reliable)
            }
        }
   }

Receiving data

Instant method session(_:didReceive:fromPeer:) indicates that new data was received from nearby peers. We’ll display the received image in the device’s image view.

func session(_ session: MCSession, didReceive data: Data, fromPeer peerID: MCPeerID) {
    if let image = UIImage(data: data){
            DispatchQueue.main.async {
                self.imageView.image = image
          }
        }
}

Conclusion

The approach to MPC video streaming described above is not the best option if you’re concerned about higher video quality or processing speed. Yet, for a simple monitoring API this works just fine. Surprisingly, sending image data instantly using send() method worked better for me than trying to employ NSStreams. However, if you’d like to challenge yourself, that might be another option to try.