LCEVC Decoder for Web (LCEVCdecJS)
LCEVC Decoder for Web (LCEVCdecJS)
Introduction
LCEVCdecJS is V-Nova's implementation of the LCEVC (MPEG-5 Part 2) decoder for web-based playback.
LCEVCdecJS is available on GitHub as a 'source available' repository.
This section provides an overview of the LCEVC decoding components for Web, included as part of the V-Nova LCEVC SDK, and provides instructions on how to integrate them into different HTML5 players.
LCEVC-enhanced HTML5 pipeline
As shown in Figure 1, the main change in the video pipeline due to enabling LCEVC decoding is to hide the original video tag region and then play back video on a new canvas region. This new canvas region and passing the MSE SourceBuffers into the LCEVCdecJS library are the main requirements of the modified video pipeline. The following sections provide further detail on these aspects as well as integration with web based video players.
The HTML5 decoding takes the following inputs:
Video segment as an MSE SourceBuffer that is also fed to the HTML Video Element;
Video element for the base decoded video picture;
A div element as destination for subtitles;
Canvas element as destination for the LCEVC-enhanced decoded video.
By using the requestAnimationFrame API, the LCEVC-enhanced decoder extracts each base video frame from the video element and renders the LCEVC-enhanced output frame on the given canvas element.
Since subtitles (e.g WebVTT) are typically rendered by the browser on the video tag, there is currently no way for the LCEVC decoder to extract that region and move it to the canvas. As a result, the solution currently offered by the decoder is to render the cues via simple, yet customizable, div rendering on top of the canvas element.
Supported browsers and platforms
The following capabilities are required in order to integrate V-Nova LCEVC HTML5 playback:
WebGL 1.0 with FrameBuffer capability. This is almost universal among current browsers and systems.
WebWorkers / WebAssembly.
WebAudio with Delay node functionality.
Typical steps taken to verify browser type and feature required including example code are described below:
Step 1: Detect the supported browser and browser version.
The supported browser and versions for different operating systems:
The supported browser and versions for Windows:
Chrome (version ≥ 57)
Firefox (version ≥ 52)
Edge (version ≥ 16)
Sample code for detecting the current browser and browser version:
Step 2: Detect the OS.
It is advised that the supported browser for the respective OS is used, as specified in Step 1.
Step 3: Check that WebGL 1.0 is available.
Create a canvas element. Note that the canvas is not added to the document itself, so it is never displayed in the browser window.
Step 4: Check if the browser supports/does not support WebWorkers.
Step 5: If all of the preceding requirements(steps) are satisfied, then proceed to the video playback. Otherwise, display an alert message to the user.
Components
The HTML5 delivery for the LCEVC SDK consists of the following sets of Javascript and Web Assembly files specific to decoding LCEVC-enhanced streams:
liblcevc_dpi.wasm: V-Nova LCEVC WebAssembly decoder library.
lcevc_dec.min.js: V-Nova LCEVC JavaScript player.
lcevc_dec_patch_xyz.js: V-Nova LCEVC JavaScript functions that are used in conjunction with the specific player being used e.g. lcevc_dec_patch_hls.js
LCEVCdecJS integration guide
This section outlines the steps and considerations needed to integrate LCEVCdecJS with any HTML5 player based on Media Source Extensions (MSE), and how to write your own patch file lcevc_dec_patch_xyz.js
. The examples given here refer to Hls.js but the principles are consistent for other players.
Overview
The general concept is that the base layer of the video (the image of the base frame) is provided by the browser's native video tag, while the LCEVC enhancement data is provided by extra data stored in the video stream. The supplied LCEVC library decodes these enhancements into a texture, then using WebGL, composites this texture on top of the original video image to create the final enhanced frame.
It is essential that the timestamp of the metadata exactly matches the timestamp of the video image, which can be achieved using JavaScript, as they are provided in different ways by the browser.
Each video player is coded differently. With some (such as Hls.js) it may be possible just to hotwire global JavaScript properties and functions, and provide the integration through a patch script. With others the only option may be to compile from source, and bundle an entire new version of the library.
The steps taken in Hls.js serve as an example and starting point to adding LCEVC support to the video technology you wish to target using a patch file. As you read through this documentation, make sure to refer to the methods outlined below, which are crucial for invoking LCEVC-enhancement decoding. Identify similar functionalities within the player you are using and apply similar modifications to ensure smooth integration with LCEVC capabilities.
1. Initialize and Check Dependencies
The initialization block sets up the environment by checking for the presence of two important dependencies, Hls.js
and LCEVCdec
within the window object. If either of these dependencies is not found, an error is thrown to ensure that the necessary prerequisites for the subsequent code execution are met.
2. Opening and Closing LCEVC
The recommended way to initialise the LCEVC object is through an attachLcevc
function that should be defined on the player object and called directly after the player has been created. We will demonstrate how to define the attachLcevc
function in further sections. This is known as creating a patch file and patching the player.
A canvas reference is passed to attachLcevc
(enhanced frames will be rendered to this canvas using WebGL) along with the configuration options (some of which can not be modified after creation).
Moving on to the integration methods for Hls.js and LCEVCdecJS, these functions establish the necessary connections and event listeners to ensure smooth interaction between the HLS player and the LCEVC decoder. They handle critical events such as media attaching, manifest loading, and buffer management to synchronize the behavior of both components effectively.
The function attachLcevc
could be defined in the patch file as an extension of the Hls.js prototype. By listening to the MEDIA_ATTACHING
and MEDIA_DETACHING
events, the player is able to create an instance of the LCEVCdecJS object in memory and remove it from memory when the player closes, seamlessly, without the host page needing any extra callbacks or scripts.
3. Appending Buffer Data
The main job of any video stream player is to parse the manifest file (.m3u8/.mpd) and manage the network, so that the correct video byte data is downloaded from the server and attached to the video player at the right time.
Each of the video technologies mentioned; e.g. Hls.js contains a function or a callback which appends incoming byte data to a MediaSourceBuffer object. At this point video data should be intercepted and a copy of the data also passed to the LCEVC decoder.
For versions of Hls.js later than 0.14.17
, the code sample below listens to buffer appending events. When video or audio/video data is appended to the buffer, it forwards this data to the LCEVC decoder for processing.
The LCEVCdecJS function LCEVCdec.appendBuffer
passes the arrayBuffer
to a WebWorker which demuxes and decodes it, then stores the extracted LCEVC data.
4. Maintaining the Profile Level and PTS Drift
An adaptive bitrate video stream can be composed of video segments with different bitrates (to accommodate changing network conditions) that are concatenated to create a seamless playback inside a single video tag. Because these segments might have slightly different start and end timestamps it is possible for the residual metadata to become misaligned with the base video layer.
The LCEVCdecJS appendBuffer
function contains a parameter for supplying a timestampOffset
, which may sometimes be supplied by the player to indicate any applied time offsets to the base video. This is helpful when the player provides different time offsets per each profile, which LCEVCdecJS is equipped to handle using appendBuffer(data, type, level, timestampOffset)
. Additionally, PTS drift is also available for cases where timestampOffset
is not sufficient, which is discussed in this section.
Some players will have an event or a callback which will indicate that the PTS offset has changed when parsing a new fragment. Sometimes they come with extra information like type of fragment, level and drift time. This extra data helps LCEVC with the synchronisation of the enhancement layer.
In Hls.js the LEVEL_PTS_UPDATED
event is used:
Whenever the player indicates that the network has changed, the settings of the video profile (collected from metadata) must be flushed. This happens whenever a level switch occurs, or the player changes video source.
Event handlers related to level switching (LEVEL_SWITCHING
, LEVEL_SWITCHED
) ensure that the LCEVC decoder receives updated information about the current playback level and any level switches. This allows the LCEVC decoder to adjust its processing parameters accordingly, optimizing the decoding process for different quality levels.
LCEVCdecJS also functions differently depending on the container format (i.e. TS
, MP4
, WEBM
) and as such it requires the container format to be set using setContainerFormat(...)
. Good place to do this is either on manifest load, or in the level switched event.
In Hls.js the following events are used:
Container format will likely be signalled differently by different players. In the code snippet below, the container format is checked against various formats (TS, WEBM, MP4), and based on the match, the corresponding container format index is passed to setContainerFormat(...)
to configure LCEVCdecJS accordingly. This ensures that the decoder operates optimally based on the specific characteristics of the media container.
API
The interface between the webpage script and the LCEVC-enhanced player is maintained by a simple API using the following properties, methods, and events.
Configurations options
Configurations parameters can be provided to lcevc_dec.min.js to create an lcevcDec
object.
debugStats
Show debug statistics including framerate, dropped frame, and other statistics as an overlay on top of the main canvas.
Default: false
renderAtDisplaySize
Use the display size instead of the video size when rendering. Useful when the display size is lower than the video size. For example, if the video is 1080p but the player size is 720p, it will render at 720p.
Default: True
logLevel
Set the LCEVC log level. Depending on the level specified, different levels of message feedback detail will be printed in the console.
Default: LCEVCdec.LogLevel.NONE
dps (Dynamic Performance Scaling)
Enable or disable dynamic performance scaling. Dynamic performance scaling monitors playback to ensure that the number of frames being dropped due to constrained system resources hasn't reached an unacceptable level. If the decoding system is struggling, LCEVC decoding will be temporarily paused with the video just displaying the base video decode.
Default: True
dynamicPerformanceScaling
See dps.
playerControls
Enables a simple player controller with some customisation options. It is used to debug the actions of a player controller. Also, it gives some information about the stream like video size, selected profile, frame rate and timestamp. It can be configured with the following parameters:
enabled:
True
to enable it, otherwiseFalse
.position: CSS style position of the controls.
offsetY: Vertical position offset of the controls.
offsetX: Horizontal position offset of the controls.
colorPlayed: Hex color of the played time bar.
colorBuffered: Hex color of the buffered video.
fullscreenElement:
HTMLElement
of the player.
Default: null
Version control
LCEVCdec.LCEVC_DEC_BUILD_DATE
Returns the build date.
Log level API
LCEVCdec.LogLevel
Log level enumeration:
NONE: 0.
ERROR: 1.
WARNING: 2.
INFO: 3.
DEBUG: 4.
VERBOSE: 5.
LCEVCdec API
new LCEVCdec.LCEVCdec(video, canvas, configOptions)
Construct a new LCEVC decoder object using the HTMLVideoElement
as the base video element, HTMLCanvasElement
as the display canvas, and the LCEVC configuration options.
LCEVCdec instance methods
lcevcDec.video get
Return the HTMLVideoElement
used as the base video element.
lcevcDec.isFullscreen get
Return True
if the video is fullscreen, otherwise False
.
lcevcDec.isLive get
Return True
if the video is live, otherwise False
.
lcevcDec.isPerformanceScalingEnabled get
Return True
if dynamic performance scaling is enabled, otherwise False
.
lcevcDec.enablePerformanceScaling(value)
Enable or disable the dynamic performance scaling.
lcevcDec.isPerformanceScalingActive get
Return True
if the performance scaling is active, otherwise False
.
lcevcDec.isLcevcEnabled get
Return True
if LCEVC decoding is enabled, otherwise False
.
lcevcDec.lcevcDataDetected get
Return True
if LCEVC data is detected in the video, otherwise False
.
lcevcDec.frameWidth get
Return the width size of the displayed frame.
lcevcDec.frameHeight get
Return the height size of the displayed frame.
lcevcDec.currentLevel get
Return the number value of the current level.
lcevcDec.firstLcevcSegmentLoaded get
Return True
if a segment with LCEVC data has been loaded, otherwise False
.
lcevcDec.aspectRatio get
Returns the value of the aspect ratio of the video.
lcevcDec.frameRate get
Returns the frame rate of the video.
lcevcDec.getConfigOption(option)
Get the value of the option
of the LCEVC configuration.
lcevcDec.setConfigOption(option, value)
Set the option
with the value
to the LCEVC configuration.
lcevcDec.logLevel get/set
Get or set the log level to the given LCEVCdec.LogLevel
(default NONE).
lcevcDec.profileIn get
Get the input color profile.
lcevcDec.setProfileIn(profileName)
Set the input color profile to the given profile name.
lcevcDec.profileOut get
Get the output color profile.
lcevcDec.setProfileOut(profileName)
Set the output color profile to the given profile name.
lcevcDec.onFullscreen(enable)
Inform the LCEVC decoder that the player has triggered a fullscreen event.
lcevcDec.clearTemporal()
Clears the LCEVC temporal buffer. This will remove the current residuals and should only be used when moving or seeking in the video or when changing the profile.
lcevcDec.setCurrentLevel(level)
Notifies LCEVC decoder that a level loaded event has been triggered passing the new level
. This ensures that LCEVC data is managed correctly upon loading a new level.
lcevcDec.setLevelSwitching(level)
Notifies LCEVC decoder that a level switching event has been triggered passing the new level
. This ensures that LCEVC data is managed correctly on the transition between levels.
lcevcDec.appendBuffer(data, type, level, timestampOffset)
Integration function only.
Receives video buffer data and sends it to the worker to detect and extract LCEVC data. The arguments are:
data: The video buffer data.
type: The type of the data, can be
'video'
,'audio'
or'audiovideo'
.level: The current ABR profile/rendition/level ID of the data.
timestampOffset: offset added to the timestamps in the fragment (defaults to 0.0).
lcevcDec.flushBuffer(startTime, endTime)
Flush the residual and drift data of the given interval of time in milliseconds.
lcevcDec.resetBuffer()
Reset the internal buffers. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.
lcevcDec.newPts(start, end, type, level, drift)
Integration function only.
Instructs LCEVC decoder that new information has been updated after parsing a fragment.
The type, level and drift are optional, but providing them helps LCEVC decoder to maintain correct synchronisation between the base video and decoded enhancement layers.
start: The start time in milliseconds of the fragment.
end: The end time in milliseconds of the fragment.
type: Type of the fragment, can be
'video'
or'audio'
.level: The level of the parsed fragment.
drift: The drift amount of time.
lcevcDec.setContainerFormat(containerFormat)
Set the container flag, whether the container is TS
, WEBM
, or MP4
. LCEVCdecJS functions differently depending on the container format. Good place to do this is either on manifest load, or in the level switched event of a player.
0: signals
TS
container to LCEVCdecJS1: signals
WEBM
container to LCEVCdecJS2: signals
MP4
container to LCEVCdecJS
lcevcDec.displayError(message)
Prints an error message in the canvas where the video is displayed.
lcevcDec.clearError()
Clears the error message of the canvas if any.
lcevcDec.close()
Closes the LCEVC decoder and releases memory in use. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.
Events API
lcevcDec.on(type, listener)
Attach an event listener of LCEVCdec.Events
type to the LCEVC decoder.
lcevcDec.off(type, listener)
Remove the event listener of the given LCEVCdec.Events
type.
Runtime events
LCEVCdec.Events.PERFORMANCE_DROPPED
Occurs when Dynamic Performance Scaling is enabled, and GPU system resources are heavily constrained.
LCEVCdec.Events.PERFORMANCE_RESTORED
Occurs after a PERFORMANCE_DROPPED event, when the GPU is able to run at optimal speed again. LCEVC enhancement decoding has been restored.
Last updated