AndroidX (ExoPlayer) with LCEVC

This article introduces how to add LCEVC decoding and rendering into androidx/media project, Google's open source Android platform media libraries specifically ExoPlayer.

Supported stream format

1. The solution utilizesMediaCodecfor decoding the base video stream so any codec supported by MediaCodec on the device can be supported for its LCEVC enhanced stream. Please consult this page to understand Android native codec support. For example, LCEVC enhanced H264 stream in local file (MPEG-TS .ts, MPEG-4 .mp4), as well as in HLS and DASH, are well supported. However, formats such as AV1 are only available on Android 10 and later, so playback of LCEVC-enhanced AV1 streams in this implementation faces the same limitation.

2. LCEVC data can be carried within the single video track using the SEI NAL unit type for MPEG codecs such as H.264, HEVC, and VVC. Alternatively, it can be contained within separate tracks, as seen in an .mp4 video source where one track carries the base video stream and another track carries the LCEVC video stream. However, it's crucial to note that the current implementation only supports the use of a single video track carried within SEI data.

Supported AndroidX (ExoPlayer) version

Since ExoPlayer version 2.13, the MediaCodecAdapterinterface has been introduced and remained stable. The implementation of the LCEVC decoder extension relies on this interface, enabling seamless integration of the solution into media applications using ExoPlayer 2.13 and later versions.

Design overview

The provided class diagram offers a high-level depiction of how LCEVC decoding and rendering are integrated into the AndroidX framework:

1. LCEVC decoding and rendering is added into ExoPlayer as a decoder extension library, by use the LCEVC decoder native library.

2. LCEVCSynchronousMediaCodecAdapter is the entry point class, which inherits the interface MediaCodecAdapterdefined in ExoPlayer core library. Its implementation is by use MediaCodec API to decode the base video stream, use LCEVC decoder native library to decode LCEVC video stream and render the final full resolution video frame.

decoder_lcevc extension architecture

How to deploy

Get decoder_lcevc extension library

For media application already migrated to AndroidX project the integration can be quite straight forward:

1. Fetch specific version of Google AndroidX libraries in your project build.gradle i.e.

   def mediaVersion = "1.0.1"
   implementation "androidx.media3:media3-exoplayer:$mediaVersion"
   implementation "androidx.media3:media3-exoplayer-dash:$mediaVersion"

2. Fetch the matching version of V-Nova decoder_lcevc library by adding following into build.gradle i.e:

   def lcevcDecoderExtensionVersion = “1.0.1.hashcode”
   implementation "androidx.media3:media3-decoder-lcevc:$lcevcDecoderExtensionVersion "

Code to enable LCEVC

Once you've followed the instructions above to fetch the AndroidX and decoder_lcevc libraries, the next step is to tell ExoPlayer to use decoder_lcevc by creating the LcevcRenderersFactory and create ExoPlayer instance:

renderersFactory = LcevcRenderersFactory(applicationContext)
player = ExoPlayer.Builder(this)
            .setRenderersFactory(renderersFactory)
            .build()