LCEVC Decoder for Web (LCEVCdecJS)

LCEVC Decoder for Web (LCEVCdecJS)

Introduction

LCEVCdecJS is V-Nova's implementation of the LCEVC (MPEG-5 Part 2) decoder for web-based playback.

LCEVCdecJS is available on GitHub as a 'source available' repository.

This section provides an overview of the LCEVC decoding components for Web, included as part of the V-Nova LCEVC SDK, and provides instructions on how to integrate them into different HTML5 players.

LCEVC-enhanced HTML5 pipeline

Figure 1: HTML5 decoding with and without LCEVC enhancement

As shown in Figure 1, the main change in the video pipeline due to enabling LCEVC decoding is to hide the original video tag region and then play back video on a new canvas region. This new canvas region and passing the MSE SourceBuffers into the LCEVCdecJS library are the main requirements of the modified video pipeline. The following sections provide further detail on these aspects as well as integration with web based video players.

The HTML5 decoding takes the following inputs:

1. Video segment as an MSE SourceBuffer that is also fed to the HTML Video Element;

2. Video element for the base decoded video picture;

3. A div element as destination for subtitles;

4. Canvas element as destination for the LCEVC-enhanced decoded video.

By using the requestAnimationFrame API, the LCEVC-enhanced decoder extracts each base video frame from the video element and renders the LCEVC-enhanced output frame on the given canvas element.

Since subtitles (e.g WebVTT) are typically rendered by the browser on the video tag, there is currently no way for the LCEVC decoder to extract that region and move it to the canvas. As a result, the solution currently offered by the decoder is to render the cues via simple, yet customizable, div rendering on top of the canvas element.

Supported browsers and platforms

The following capabilities are required in order to integrate V-Nova LCEVC HTML5 playback:

• WebGL 1.0 with FrameBuffer capability. This is almost universal among current browsers and systems.

• WebWorkers / WebAssembly.

• WebAudio with Delay node functionality.

Typical steps taken to verify browser type and feature required including example code are described below:

Step 1: Detect the supported browser and browser version.

The supported browser and versions for different operating systems:

Windows

The supported browser and versions for Windows:

• Chrome (version ≥ 57)

• Firefox (version ≥ 52)

• Edge (version ≥ 16)

MacOS

The supported browser and versions for MacOS:

• Chrome (version ≥ 57)

• Firefox (version ≥ 52)

• Safari (version ≥ 11)

Ubuntu

The supported browser and versions for Ubuntu 18.04 LTS:

• Chrome version 92.0.4515.131

• Firefox version 94.0

Specified or newer versions are supported.

Sample code for detecting the current browser and browser version:

Sample Code: Detect the current browser and browser version

var nVer = navigator.appVersion;
var nAgt = navigator.userAgent;
var browserName = navigator.appName;
var fullVersion, verOffset, majorVersion;

// In Chrome, the true version is after "Chrome".
if ((verOffset = nAgt.indexOf('Chrome')) !== -1)
{
	browserName = 'Chrome';
	fullVersion = nAgt.substring(verOffset + 7);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Safari, the true version is after "Safari" or after "Version".
else if ((verOffset = nAgt.indexOf('Safari')) !== -1)
{
	browserName = 'Safari';
	fullVersion = nAgt.substring(verOffset + 7);
	if ((verOffset = nAgt.indexOf('Version')) !== -1)
		fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Firefox, the true version is after "Firefox".
else if ((verOffset = nAgt.indexOf('Firefox')) != -1)
{
	browserName = 'Firefox';
	fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
//Edge
else if ((verOffset = nAgt.indexOf('Edge')) != -1)
{
	browserName = 'Edge';
	fullVersion = nAgt.substring(verOffset + 5);
	majorVersion = parseInt('' + fullVersion, 10);
}
else
{
	//Browser not supported.
}

Step 2: Detect the OS.

It is advised that the supported browser for the respective OS is used, as specified in Step 1.

Sample Code: Detect the operating system

var OSName;
if (navigator.appVersion.indexOf('Win') != -1) OSName = 'Windows';
if (navigator.appVersion.indexOf('Mac') != -1) OSName = 'MacOS';
if (navigator.appVersion.indexOf('X11') != -1) OSName = 'Unix';
if (navigator.appVersion.indexOf('Linux') != -1) OSName = 'Linux';

Step 3: Check that WebGL 1.0 is available.

Create a canvas element. Note that the canvas is not added to the document itself, so it is never displayed in the browser window.

Sample Code: Check for WebGL support

var canvas = document.createElement('canvas');

// Get WebGLRenderingContext from canvas element.
var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');

if(gl && gl instanceof WebGLRenderingContext) {
	// Congratulations! Your browser supports WebGL.
} else {
	// Your browser may not support WebGL.
}

Step 4: Check if the browser supports/does not support WebWorkers.

Sample Code: Check for WebWorker support

if (window.Worker) {
	// Congratulations! Your browser supports WebWorkers.”
} else {
	// Your browser does not support WebWorkers.
}

Step 5: If all of the preceding requirements(steps) are satisfied, then proceed to the video playback. Otherwise, display an alert message to the user.

Components

The HTML5 delivery for the LCEVC SDK consists of the following sets of Javascript and Web Assembly files specific to decoding LCEVC-enhanced streams:

• liblcevc_dpi.wasm: V-Nova LCEVC WebAssembly decoder library.

• lcevc_dec.min.js: V-Nova LCEVC JavaScript player.

• lcevc_dec_patch_xyz.js: V-Nova LCEVC JavaScript functions that are used in conjunction with the specific player being used e.g. lcevc_dec_patch_hls.js

LCEVCdecJS integration guide

This section outlines the steps and considerations needed to integrate LCEVCdecJS with any HTML5 player based on Media Source Extensions (MSE), and how to write your own patch file lcevc_dec_patch_xyz.js. The examples given here refer to Hls.js but the principles are consistent for other players.

Overview

The general concept is that the base layer of the video (the image of the base frame) is provided by the browser's native video tag, while the LCEVC enhancement data is provided by extra data stored in the video stream. The supplied LCEVC library decodes these enhancements into a texture, then using WebGL, composites this texture on top of the original video image to create the final enhanced frame.

It is essential that the timestamp of the metadata exactly matches the timestamp of the video image, which can be achieved using JavaScript, as they are provided in different ways by the browser.

Each video player is coded differently. With some (such as Hls.js) it may be possible just to hotwire global JavaScript properties and functions, and provide the integration through a patch script. With others the only option may be to compile from source, and bundle an entire new version of the library.

The steps taken in Hls.js serve as an example and starting point to adding LCEVC support to the video technology you wish to target using a patch file. As you read through this documentation, make sure to refer to the methods outlined below, which are crucial for invoking LCEVC-enhancement decoding. Identify similar functionalities within the player you are using and apply similar modifications to ensure smooth integration with LCEVC capabilities.

1. Initialize and Check Dependencies

The initialization block sets up the environment by checking for the presence of two important dependencies, Hls.js and LCEVCdec within the window object. If either of these dependencies is not found, an error is thrown to ensure that the necessary prerequisites for the subsequent code execution are met.

const { Hls } = window;
if (!Hls) {
  throw new TypeError('Hls.js needs to be loaded before patch can be applied.');
}

const { LCEVCdec } = window;
if (!LCEVCdec) {
  throw new TypeError('LCEVC needs to be loaded before patch can be applied.');
}

2. Opening and Closing LCEVC

The recommended way to initialise the LCEVC object is through an attachLcevc function that should be defined on the player object and called directly after the player has been created. We will demonstrate how to define the attachLcevc function in further sections. This is known as creating a patch file and patching the player.

// Get reference to video that LCEVCdecJS will use for input
video = document.getElementById('video');

// Get reference to canvas that LCEVCdecJS will use for rendering
canvas = document.getElementById('canvas');

// Set LCEVC config options
const lcevcDecConfig = {
  debugStats: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'top',
  }
};

// Create LCEVCdecJS instance within Hls.js object
hls.attachLcevc(canvas, lcevcDecConfig);

A canvas reference is passed to attachLcevc (enhanced frames will be rendered to this canvas using WebGL) along with the configuration options (some of which can not be modified after creation).

Moving on to the integration methods for Hls.js and LCEVCdecJS, these functions establish the necessary connections and event listeners to ensure smooth interaction between the HLS player and the LCEVC decoder. They handle critical events such as media attaching, manifest loading, and buffer management to synchronize the behavior of both components effectively.

The function attachLcevc could be defined in the patch file as an extension of the Hls.js prototype. By listening to the MEDIA_ATTACHING and MEDIA_DETACHING events, the player is able to create an instance of the LCEVCdecJS object in memory and remove it from memory when the player closes, seamlessly, without the host page needing any extra callbacks or scripts.

// attachLcevc connects the instanced Hls.js player to the LCEVC decoder
Hls.prototype.attachLcevc = function (canvas, config) {
  this.canvas = canvas;
  this.LCEVCdecConfig = config;

  // attachMedia connects the new HLS player to the V-Nova's
  // LCEVC decoder and passes the objects to it.
  
  // MEDIA_ATTACHING instances the LCEVC decoder, passing the video tag,
  // the canvas tag, and the LCEVC config
    this.on(Hls.Events.MEDIA_ATTACHING, function mediaAttaching(_, data) {
    this.LCEVCdec = new LCEVCdec.LCEVCdec(data.media, this.canvas, this.LCEVCdecConfig);
  });

  // MEDIA_DETACHING removes the LCEVC components from memory
  this.on(Hls.Events.MEDIA_DETACHING, (event) => {
    if (this.LCEVCdec) {
      this.LCEVCdec.close();
      this.LCEVCdec = null;
    }
  });

  // ...
  // Further methods for appending buffer data, handling profile
  // switching events, etc, should be placed here within the
  // same attachLcevc function.
}

3. Appending Buffer Data

The main job of any video stream player is to parse the manifest file (.m3u8/.mpd) and manage the network, so that the correct video byte data is downloaded from the server and attached to the video player at the right time.

Each of the video technologies mentioned; e.g. Hls.js contains a function or a callback which appends incoming byte data to a MediaSourceBuffer object. At this point video data should be intercepted and a copy of the data also passed to the LCEVC decoder.

For versions of Hls.js later than 0.14.17, the code sample below listens to buffer appending events. When video or audio/video data is appended to the buffer, it forwards this data to the LCEVC decoder for processing.

this.on(Hls.Events.BUFFER_APPENDING, function bufferFlushing(_, data) {
  if ((data.type === 'video' || data.type === 'audiovideo') && this.LCEVCdec) {
    this.LCEVCdec.appendBuffer(
      data.data,
      data.type,
      data.frag.level
    );
  }
});

The LCEVCdecJS function LCEVCdec.appendBuffer passes the arrayBuffer to a WebWorker which demuxes and decodes it, then stores the extracted LCEVC data.

4. Maintaining the Profile Level and PTS Drift

An adaptive bitrate video stream can be composed of video segments with different bitrates (to accommodate changing network conditions) that are concatenated to create a seamless playback inside a single video tag. Because these segments might have slightly different start and end timestamps it is possible for the residual metadata to become misaligned with the base video layer.

The LCEVCdecJS appendBuffer function contains a parameter for supplying a timestampOffset, which may sometimes be supplied by the player to indicate any applied time offsets to the base video. This is helpful when the player provides different time offsets per each profile, which LCEVCdecJS is equipped to handle using appendBuffer(data, type, level, timestampOffset). Additionally, PTS drift is also available for cases where timestampOffset is not sufficient, which is discussed in this section.

Some players will have an event or a callback which will indicate that the PTS offset has changed when parsing a new fragment. Sometimes they come with extra information like type of fragment, level and drift time. This extra data helps LCEVC with the synchronisation of the enhancement layer.

In Hls.js the LEVEL_PTS_UPDATED event is used:

// Add level PTS update event to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_PTS_UPDATED, function levelPTSUpdated(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.newPts(data.start, data.end, data.level, data.type, data.drift);
  }
});

Whenever the player indicates that the network has changed, the settings of the video profile (collected from metadata) must be flushed. This happens whenever a level switch occurs, or the player changes video source.

Event handlers related to level switching (LEVEL_SWITCHING, LEVEL_SWITCHED) ensure that the LCEVC decoder receives updated information about the current playback level and any level switches. This allows the LCEVC decoder to adjust its processing parameters accordingly, optimizing the decoding process for different quality levels.

LCEVCdecJS also functions differently depending on the container format (i.e. TS, MP4, WEBM) and as such it requires the container format to be set using setContainerFormat(...). Good place to do this is either on manifest load, or in the level switched event.

In Hls.js the following events are used:

// Buffer flushing events clears residuals when buffer flushed.
this.on(Hls.Events.BUFFER_FLUSHING, function bufferFlushing(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.flushBuffer(data.startOffset, data.endOffset);
  }
});

// Send level switching event data level to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_SWITCHING, function levelSwitching(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.setLevelSwitching(data.level);
  }
});

// Send level switched event data level to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_SWITCHED, function levelSwitchted(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.setCurrentLevel(data.level);

    if (this.streamController
      && this.streamController.fragCurrent
      && this.streamController.fragCurrent.initSegment
      && this.streamController.fragCurrent.relurl) {
      if (this.streamController.fragCurrent.relurl.includes('.mp4')) {
        this.LCEVCdec.setContainerFormat(2);
      }
    }
  }
});

Container format will likely be signalled differently by different players. In the code snippet below, the container format is checked against various formats (TS, WEBM, MP4), and based on the match, the corresponding container format index is passed to setContainerFormat(...) to configure LCEVCdecJS accordingly. This ensures that the decoder operates optimally based on the specific characteristics of the media container.

if (containerFormat.includes('video/ts')) {
    this.LCEVCdec.setContainerFormat(0);
} else if (containerFormat.includes('video/webm')) {
    this.LCEVCdec.setContainerFormat(1);
} else if (containerFormat.includes('video/mp4')) {
    this.LCEVCdec.setContainerFormat(2);
}

API

The interface between the webpage script and the LCEVC-enhanced player is maintained by a simple API using the following properties, methods, and events.

Configurations options

Configurations parameters can be provided to lcevc_dec.min.js to create an lcevcDec object.

const LCEVCdecDefaultConfig = {
  debugStats: false,
  renderAtDisplaySize: true,
  logLevel: 0,
  dps: true,
  dynamicPerformanceScaling: true,
  iOSFallback: false,
  logo: true,
  drawLogo: true,
  loop: false,
  poster: true,
  ebmlVersion: '3.0.0',
  mp4boxVersion: '0.5.2',
  ebmlLoadOnStartup: false,
  mp4boxLoadOnStartup: true,

  playerControls: {
    controlsID: 'player-controls',
    enabled: false,
    playButton: true,
    seekButton: false,
    volumeButton: false,
    fullscreenButton: true,
    fullscreenButtonPos: 'left',
    toggleButton: false,
    toggleButtonPos: 'right',
    position: 'bottom',
    offsetY: 24,
    offsetX: 0,
    colorPlayed: 0x5e7ebd,
    colorBuffered: 0xadadad,
    colorUnplayed: 0x5e5e5e,
    fullscreenElement: null
  }
};

debugStats

Show debug statistics including framerate, dropped frame, and other statistics as an overlay on top of the main canvas.

Default: false

renderAtDisplaySize

Use the display size instead of the video size when rendering. Useful when the display size is lower than the video size. For example, if the video is 1080p but the player size is 720p, it will render at 720p.

Default: True

logLevel

Set the LCEVC log level. Depending on the level specified, different levels of message feedback detail will be printed in the console.

Default: LCEVCdec.LogLevel.NONE

dps (Dynamic Performance Scaling)

Enable or disable dynamic performance scaling. Dynamic performance scaling monitors playback to ensure that the number of frames being dropped due to constrained system resources hasn't reached an unacceptable level. If the decoding system is struggling, LCEVC decoding will be temporarily paused with the video just displaying the base video decode.

Default: True

dynamicPerformanceScaling

See dps.

playerControls

Enables a simple player controller with some customisation options. It is used to debug the actions of a player controller. Also, it gives some information about the stream like video size, selected profile, frame rate and timestamp. It can be configured with the following parameters:

• enabled: True to enable it, otherwise False.

• position: CSS style position of the controls.

• offsetY: Vertical position offset of the controls.

• offsetX: Horizontal position offset of the controls.

• colorPlayed: Hex color of the played time bar.

• colorBuffered: Hex color of the buffered video.

• fullscreenElement: HTMLElement of the player.

Default: null

Version control

LCEVCdec.LCEVC_DEC_BUILD_DATE

Returns the build date.

Log level API

LCEVCdec.LogLevel

Log level enumeration:

• NONE: 0.

• ERROR: 1.

• WARNING: 2.

• INFO: 3.

• DEBUG: 4.

• VERBOSE: 5.

LCEVCdec API

new LCEVCdec.LCEVCdec(video, canvas, configOptions)

Construct a new LCEVC decoder object using the HTMLVideoElement as the base video element, HTMLCanvasElement as the display canvas, and the LCEVC configuration options.

LCEVCdec instance methods

lcevcDec.video get

Return the HTMLVideoElement used as the base video element.

lcevcDec.isFullscreen get

Return True if the video is fullscreen, otherwise False.

lcevcDec.isLive get

Return True if the video is live, otherwise False.

lcevcDec.isPerformanceScalingEnabled get

Return True if dynamic performance scaling is enabled, otherwise False.

lcevcDec.enablePerformanceScaling(value)

Enable or disable the dynamic performance scaling.

lcevcDec.isPerformanceScalingActive get

Return True if the performance scaling is active, otherwise False.

lcevcDec.isLcevcEnabled get

Return True if LCEVC decoding is enabled, otherwise False.

lcevcDec.lcevcDataDetected get

Return True if LCEVC data is detected in the video, otherwise False.

lcevcDec.frameWidth get

Return the width size of the displayed frame.

lcevcDec.frameHeight get

Return the height size of the displayed frame.

lcevcDec.currentLevel get

Return the number value of the current level.

lcevcDec.firstLcevcSegmentLoaded get

Return True if a segment with LCEVC data has been loaded, otherwise False.

lcevcDec.aspectRatio get

Returns the value of the aspect ratio of the video.

lcevcDec.frameRate get

Returns the frame rate of the video.

lcevcDec.getConfigOption(option)

Get the value of the option of the LCEVC configuration.

lcevcDec.setConfigOption(option, value)

Set the option with the value to the LCEVC configuration.

lcevcDec.logLevel get/set

Get or set the log level to the given LCEVCdec.LogLevel (default NONE).

lcevcDec.profileIn get

Get the input color profile.

lcevcDec.setProfileIn(profileName)

Set the input color profile to the given profile name.

lcevcDec.profileOut get

Get the output color profile.

lcevcDec.setProfileOut(profileName)

Set the output color profile to the given profile name.

lcevcDec.onFullscreen(enable)

Inform the LCEVC decoder that the player has triggered a fullscreen event.

lcevcDec.clearTemporal()

Clears the LCEVC temporal buffer. This will remove the current residuals and should only be used when moving or seeking in the video or when changing the profile.

lcevcDec.setCurrentLevel(level)

Notifies LCEVC decoder that a level loaded event has been triggered passing the new level. This ensures that LCEVC data is managed correctly upon loading a new level.

lcevcDec.setLevelSwitching(level)

Notifies LCEVC decoder that a level switching event has been triggered passing the new level. This ensures that LCEVC data is managed correctly on the transition between levels.

lcevcDec.appendBuffer(data, type, level, timestampOffset)

Integration function only.

Receives video buffer data and sends it to the worker to detect and extract LCEVC data. The arguments are:

• data: The video buffer data.

• type: The type of the data, can be 'video', 'audio' or 'audiovideo'.

• level: The current ABR profile/rendition/level ID of the data.

• timestampOffset: offset added to the timestamps in the fragment (defaults to 0.0).

lcevcDec.flushBuffer(startTime, endTime)

Flush the residual and drift data of the given interval of time in milliseconds.

lcevcDec.resetBuffer()

Reset the internal buffers. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

lcevcDec.newPts(start, end, type, level, drift)

Integration function only.

Instructs LCEVC decoder that new information has been updated after parsing a fragment.

The type, level and drift are optional, but providing them helps LCEVC decoder to maintain correct synchronisation between the base video and decoded enhancement layers.

• start: The start time in milliseconds of the fragment.

• end: The end time in milliseconds of the fragment.

• type: Type of the fragment, can be 'video' or 'audio'.

• level: The level of the parsed fragment.

• drift: The drift amount of time.

lcevcDec.setContainerFormat(containerFormat)

Set the container flag, whether the container is TS, WEBM, or MP4. LCEVCdecJS functions differently depending on the container format. Good place to do this is either on manifest load, or in the level switched event of a player.

• 0: signals TS container to LCEVCdecJS

• 1: signals WEBM container to LCEVCdecJS

• 2: signals MP4 container to LCEVCdecJS

lcevcDec.displayError(message)

Prints an error message in the canvas where the video is displayed.

lcevcDec.clearError()

Clears the error message of the canvas if any.

lcevcDec.close()

Closes the LCEVC decoder and releases memory in use. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

Events API

lcevcDec.on(type, listener)

Attach an event listener of LCEVCdec.Events type to the LCEVC decoder.

lcevcDec.off(type, listener)

Remove the event listener of the given LCEVCdec.Events type.

Runtime events

LCEVCdec.Events.PERFORMANCE_DROPPED

Occurs when Dynamic Performance Scaling is enabled, and GPU system resources are heavily constrained.

lcevcDec.on(LCEVCdec.Events.PERFORMANCE_DROPPED, function(){
    console.log('Performance Dropped');
}

LCEVCdec.Events.PERFORMANCE_RESTORED

Occurs after a PERFORMANCE_DROPPED event, when the GPU is able to run at optimal speed again. LCEVC enhancement decoding has been restored.

lcevcDec.on(LCEVCdec.Events.PERFORMANCE_RESTORED, function(){
    console.log('Performance Restored');
}