VLCKit with LCEVC

1 Introduction

This section provides an overview of how to build and use VLCKit, how VLCKit uses Apple’s VideoToolbox and how LCEVC decoding can be achieved inside VLCKit.

A general overview of VLCKit and its capabilities is available at https://wiki.videolan.org/VLCKit.

The LCEVC-enabled VLCKit has been extensively tested and validated by third party test house Kineton to ensure that the addition of LCEVC to the player maintained all of its functionality and performance. The results of the tests are available here.

2 Integrating VLCKit

The VLCKit repository can be found in VideoLan’s Gitlab. The vanilla VLCKit is available through CocoaPods, Carthage and you can build it locally on your machine. The VLCKit that is commercially used is not on master branch yet, it’s stored in branch 3.0. For the purposes of this document, we will use the 3.0 branch and build it locally on the machine.

2-1 Building VLCKit

Before starting the build, make sure you have enough storage space (~6GB).

2-1-1 Vanilla VLCKit

By going into the root directory of VLCKit, you can start the build process with the following commands:

./buildMobileVLCKit.sh -f

You can also specify which architecture you want to build VLCKit for:

./buildMobileVLCKit.sh -a aarch64

Once the build finishes, you can find the MobileVLCKit.framework in the build directory. You can link this framework directly inside XCode:

1. Go to Project > Targets > ${YOUR_APPLICATION} > General and add framework to Frameworks, Libraries and Embedded content section.

2. Make sure Framework Search Paths and Header Search Paths in Project > Targets > ${YOUR_APPLICATION} > Build Settings can find the .framework and its Headers directory.

For more information on how to integrate frameworks, check Apple’s documentations.

2-1-2 VLCKit + LCEVC

VLCKit has a mechanism for automatically applying patches to libvlc (VLC’s core) before starting the build. The required changes for applying LCEVC to VLCKit consist of multiple patches and a static library binary. You just need to copy the patches into Resources/MobileVLCKit/patches folder and start the build. Once the build is finished, integrate the framework just like before.

You also need to embed the lcevc_dec_videotoolbox.a binary into your application. Same as frameworks:

1. Go to Project > Targets > ${YOUR_APPLICATION} > General and add framework to Frameworks, Libraries and Embedded content section.

2. Make sure Library Search Paths in Project > Targets > ${YOUR_APPLICATION} > Build Settings can find the lcevc_dec_videotoolbox.a file.

VLCKit should now be able to decode LCEVC streams.

2-2 Using VLCKit

In order to use VLCKit, first you need an instance of VLCMediaPlayer. Then an UIView to set as render target and a VLCMedia instance which will load the asset.

PlayerViewController.swift

class PlayerViewController: UIViewController {
    var url: URL!
    let player = VLCMediaPlayer()
    
    override func viewDidLoad() {
        super.viewDidLoad()
        setupMediaLibrary()
    }
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        player.play()
    }
    
    private func setupMediaLibrary() {
        player.delegate = self
        player.drawable = view
        player.media = VLCMedia(url: url)
    }
}

extension PlayerViewController: VLCMediaPlayerDelegate {
    func mediaPlayerStateChanged(_ aNotification: Notification!) {
        if player.state == .stopped {
            self.dismiss(animated: true, completion: nil)
        }
    }
}

For more information on how to use VLCKit you can check their examples, documentations and VLC-iOS.

3 Apple’s VideoToolbox

In order to work with Apple’s VideoToolbox Decoder module (VTDecompressionSession), the stream must be demuxed into CMSampleBuffers and fed into VTDecompressionSession for decompression. In response VTDecompressionSession will return CVPixelBuffers for given CMSampleBuffers in decoding order. After reordering CVPixelBuffers to presentation order, they’re ready for presentation on screen.

3-1 VLCKit videotoolbox module

All the logics mentioned earlier are encapsulated into a single module named videtoolbox in VLCs ecosystem. The process is shown below:

3-2 VLCKit + LCEVC

In order to apply LCEVC decoding, we need to extract LCEVC payload before decompression, and apply LCEVC once decompression is done and frames are ordered by presentation. The process is shown below: