ExoPlayer with LCEVC

Introduction
This article describes the V-Nova LCEVC ExoPlayer Android media player.
ExoPlayer is a Google-maintained project under Apache License 2.0. LCEVC ExoPlayer is a derived work by V-Nova Ltd, under commercial license, that adds support for playing back streams enhanced with MPEG-5 Part 2 LCEVC, in the form of an ExoPlayer extension.
ExoPlayer is an Android library for media playback, supporting features not currently provided by the standard Android MediaPlayer, including Dynamic Adaptive Streaming over HTTP (DASH), SmoothStreaming and Common Encryption. It is designed to be easy to customize and extend, allowing many components to be replaced with custom implementations. In particular the extensions are provided as a mechanism to add custom implementations of existing or new codecs.
ExoPlayer is widely used as a playback library in Android media player apps, including premium video apps.
The official project page is http://google.github.io/ExoPlayer, which provides information about the project and the features supported by the media player library.
Note: ExoPlayer is an open source project that, although developed primarily by Google, is not part of the Android framework, and is distributed separately from the Android SDK.
The LCEVC-enabled ExoPlayer has been extensively tested and validated by third party test house Kineton to ensure that the addition of LCEVC to the player maintained all of its functionality and performance. The results of the tests are available here.
​
Versioning
LCEVC ExoPlayer is currently based on Google’s ExoPlayer release r2.11.1. The SDK uses the following internal variables to signal git's properties related to the V-Nova project:
GitTag
GitDate
GitHash
BUILD_TYPE
The git tag, in particular, specifies the original ExoPlayer release on which the project is based (“r2.11.1”), the support for LCEVC, and the version identification of the LCEVC ExoPlayer project.
The above properties are carried in the file “project.properties” in the project root directory.
The V-Nova LCEVC Extension is dependent on the Decoder Integration (DIL) library, with the following metadata:
DIL GitBranch
DIL GitDate
DIL GitHash
The above DIL-related properties are carried separately in the file “dil.properties” in the project root directory.
All of the above versioning details are imported from the gradle-based build system and available programmatically from the “BuildConfig” class. They are printed in the logcat from the LCEVC ExoPlayer-based demo app, and can also be viewed as pop-up from the demo user interface, by long pressing on the “GO” button.
Design
Design overview
LCEVC ExoPlayer is the V-Nova derivative work from Google’s ExoPlayer, branched from the release indicated above, in which an LCEVC decoder, using the V-Nova's LCEVC DIL library, has been added as an extension to the project. This extension wraps the Android standard MediaCodec renderer, and enhances it according to the LCEVC specification by means of the DIL library. The extension also provides a pass-through functionality; therefore, it can render native video only if LCEVC enhancement bitstream is not found in the stream.
The LCEVC extension, composed of the Android MediaCodec class plus the DIL, is therefore functionally a replacement of the standard MediaCodec renderer. Since the integration is at the MediaCodec level, all features provided by ExoPlayer are generally maintained. Thus, for example, support for the different ABR protocols (HLS, DASH, SS) as well as Common Encryption, implemented in the core ExoPlayer library, continue to exist in the LCEVC based player.
The following figure shows the design of the unmodified versus LCEVC integrated video pipeline which can be applied to ExoPlayer.
Figure 1 Unmodified vs DIL integrated LCEVC video pipeline 
In LCEVC ExoPlayer, both the original renderer, based on the MediaCodec class, and the LCEVC Extension are registered as video renderers, with MediaCodec the default renderer.
In a generic application, the selection of the LCEVC extension renderer over the default one is determined by the extensionRendererMode parameter, set as EXTENSION_RENDERER_MODE_PREFER, as passed to the constructor of the DefaultRenderersFactory class.
In the demo application, this parameter is set in the intent that is passed from the SampleChooserActivity to the PlayerActivity. This intent, in turn, is set from the UI in the Context Menu. Setting extensionRendererMode as EXTENSION_RENDERER_MODE_PREFER (mode “prefer”) is required for LCEVC content.
The above property may be left unset for non-LCEVC-enhanced content. By doing so the content will be played through the default ExoPlayer pipeline, with no difference in behaviour, compared to the vanilla ExoPlayer.
Components
The V-Nova LCEVC integration affects the three core modules in the following manner:
1.
Minor changes to the ExoPlayer libraries “library-core” and “library-ui”;
2.
Extensions: Added an LCEVC extension for the LCEVC video renderer;
3.
Added UI components to manually enter a url or select a local file; extended the list of predefined urls to add LCEVC-enhanced test content (in the MISC section of the ExoPlayer interface);
Important: LCEVC ExoPlayer requires minimum Android version = 5.0 (Lollipop), corresponding to API Level >= 21.
Application API
In the case of LCEVC-enhanced content, the application shall set the use of the LCEVC renderer in place of the default renderer by means of the extensionRendererMode parameter set to EXTENSION_RENDERER_MODE_PREFER when the player activity constructs the DefaultRenderersFactory object.
Modified Source Code
As a requirement of the Apache License 2.0 of the ExoPlayer project V-Nova publishes a list of files from the original ExoPlayer source tree that have been modified by V-Nova along with a patch showing these changes. The following diagram shows which elements of ExoPlayer are modified to integrate LCEVC.
Figure 2 Modified source code
 
Usage
ExoPlayer UI
The original sample application that is provided with the ExoPlayer project has been modified, by adding a top section, SampleChooserActivity class, that provides the ability to enter a URL, or to select a local file and also get version information. Usage of the sample application is very straightforward as shown by the following screenshots:
Figure 3 Selection UI of the demo app
627b1e16f9e5d3c73e2055cf36fe0672
Figure 4 Playback controls of the demo app
Playback capability
The app can play:
1.
Local files, by selecting the GO button with an empty URL field
2.
A URL entered in the URL field (see http://google.github.io/ExoPlayer/supported-formats.html for the supported protocols/formats, excluding LCEVC)
Note: instead of manually entering long URLs, the following adb command line can be used: "adb shell input text http://..."
1.
The predefined ExoPlayer test URLs located in the MISC section of the ExoPlayer interface (read from the asset file media.exolist.json).
2.
URLs from an additional json formatted list, if found on the device’s filesystem in:
/sdcard/local.exolist.json
Statistics
The UI displays the following parameters:
Header info:
playWhenReady: false/true (false when paused, true when playing)
playbackState: buffering/ready (buffering or ready)
For each video and audio:
codec mime type
id: id of the selected representation (= ABR profile from DASH/HLS)
(video) r: resolution
(audio) hz: sampling freq., ch: n. of audio channels
rb: rendered buffers (i.e., audio or video frames)
sb: skipped buffers (e.g., because of a seek)
db: dropped buffers (because decoded "late")
mcdb: most consecutive dropped buffers (max number of adjacent frames dropped)
One button per component (video/audio/subtitle, if present)
if selected, displays a popup to enable/disable the track and select among the available ABR profiles. The relative bitrate is also displayed.
Version Information
To display the versioning details of both the LCEVC ExoPlayer project and the DIL that it embeds, long press the “GO” button to open the Version Information popup.
Development
Android Studio deliverable
LCEVC ExoPlayer is provided by V-Nova as SDK in the form of a set of Android Archives (AAR), ready to be imported in an existing ExoPlayer based project, as a drop-in replacement of the ones from the original (vanilla) repository, with all components under a commercial license that has been separately signed. The set contains the same modules as the vanilla ExoPlayer project, except for the following Android Studio modules:
1.
library-core with LCEVC modifications in object code (.AAR)
2.
library-ui with LCEVC modifications in object code (.AAR)
3.
additional LCEVC renderer extension library in object code (.AAR)
4.
modified sample player application in source code
A step by step guide on importing the AAR libraries into a working demo project are also provided in the SDK archive.
Supported ABIs
The ExoPlayer core library is an application library. As such its object code is Dalvik byte code ABI independent. The LCEVC renderer extension library supports:
ABI
armeabi-v7a
arm64-v8a
x86
x86_64