Decoder Integration Layer for Web (DIL.js)

Introduction

This section provides an overview of the LCEVC decoding components for Web, included as part of the V-Nova LCEVC SDK, and provides instructions on how to integrate them into different HTML5 players.

LCEVC-enhanced HTML5 pipeline

As shown in Figure 1, the main change in the video pipeline due to enabling LCEVC decoding is to hide the original video tag region and then play back video on a new canvas region. This new canvas region and passing the MSE SourceBuffers into the LcevcDil.js module are the main requirements of the modified video pipeline. The following sections provide further detail on these aspects as well as integration with web based video players.

The HTML5 decoding takes the following inputs:

  1. Video segment as an MSE SourceBuffer that is also fed to the HTML Video Element;

  2. Video element for the base decoded video picture;

  3. A div element as destination for subtitles;

  4. Canvas element as destination for the LCEVC-enhanced decoded video.

By using the requestAnimationFrame API, the LCEVC-enhanced decoder extracts each video frame from the video element and renders the LCEVC-enhanced output frame on the given canvas element.

Since subtitles (e.g WebVTT) are typically rendered by the browser on the video tag, there is currently no way for the LCEVC decoder to extract that region and move it to the canvas. As a result, the solution currently offered by the decoder is to render the cues via simple, yet customizable, div rendering on top of the canvas element.

Supported browsers and platforms

The following capabilities are required in order to integrate V-Nova LCEVC HTML5 playback:

  • WebGL 1.0 with FrameBuffer capability. This is almost universal among current browsers and systems.

  • WebWorkers / WebAssembly.

  • WebAudio with Delay node functionality.

Typical steps taken to verify browser type and feature required including example code are described below:

Step 1: Detect the supported browser and browser version.

The supported browser and versions for Windows:

  • Chrome (version ≥ 57)

  • Firefox (version ≥ 52)

  • Edge (version ≥ 16)

The supported browser and versions for MacOS:

  • Chrome (version ≥ 57)

  • Firefox (version ≥ 52)

  • Safari (version ≥ 11)

Browsers tested on Ubuntu 18.04 LTS:

  • Chrome version 92.0.4515.131

  • Firefox version 94.0

Sample code:

var nVer = navigator.appVersion;
var nAgt = navigator.userAgent;
var browserName = navigator.appName;
var fullVersion, verOffset, majorVersion;

// In Chrome, the true version is after "Chrome".
if ((verOffset = nAgt.indexOf('Chrome')) !== -1)
{
	browserName = 'Chrome';
	fullVersion = nAgt.substring(verOffset + 7);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Safari, the true version is after "Safari" or after "Version".
else if ((verOffset = nAgt.indexOf('Safari')) !== -1)
{
	browserName = 'Safari';
	fullVersion = nAgt.substring(verOffset + 7);
	if ((verOffset = nAgt.indexOf('Version')) !== -1)
		fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Firefox, the true version is after "Firefox".
else if ((verOffset = nAgt.indexOf('Firefox')) != -1)
{
	browserName = 'Firefox';
	fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
//Edge
else if ((verOffset = nAgt.indexOf('Edge')) != -1)
{
	browserName = 'Edge';
	fullVersion = nAgt.substring(verOffset + 5);
	majorVersion = parseInt('' + fullVersion, 10);
}
else
{
	//Browser not supported.
}

Step 2: Detect the OS.

It is advised that the supported browser for the respective OS is used, as specified in Step 1. 

var OSName;
if (navigator.appVersion.indexOf('Win') != -1) OSName = 'Windows';
if (navigator.appVersion.indexOf('Mac') != -1) OSName = 'MacOS';
if (navigator.appVersion.indexOf('X11') != -1) OSName = 'Unix';
if (navigator.appVersion.indexOf('Linux') != -1) OSName = 'Linux';

Step 3: Check that WebGL 1.0 is available.

Create a canvas element. Note that the canvas is not added to the document itself, so it is never displayed in the browser window.

var canvas = document.createElement('canvas');

// Get WebGLRenderingContext from canvas element.
var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');

if(gl && gl instanceof WebGLRenderingContext) {
	// Congratulations! Your browser supports WebGL.
} else {
	// Your browser may not support WebGL.
}

Step 4: Check if the browser supports/does not support WebWorkers.

if (window.Worker) {
	// Congratulations! Your browser supports WebWorkers.”
} else {
	// Your browser does not support WebWorkers.
}

Step 5: If all of the preceding requirements(steps) are satisfied, then proceed to the video playback. Otherwise, display an alert message to the user.

Components

The HTML5 delivery for the LCEVC SDK consists of the following sets of Javascript and Web Assembly files specific to decoding LCEVC-enhanced streams:

  • liblcevc_dpi.wasm: V-Nova LCEVC WebAssembly decoder library.

  • lcevc_dil.min.js: V-Nova LCEVC JavaScript player.

  • lcevc_dil_patch_xyz.js: V-Nova LCEVC JavaScript functions that are used in conjunction with the specific player being used e.g. lcevc_dil_patch_shaka.js

dil.js integration guide

This section outlines the steps and considerations needed to integrate dil.js with any HTML5 player based on Media Source Extensions (MSE), and how to write your own patch file. The examples given here refer to hls.js but the principles are consistent for other players.

Overview

The general concept is that the base layer of the video (the image of the frame) is provided by the browser's native video tag, while the LCEVC enhancement data is provided by extra data stored in the video stream. The supplied LCEVC library decodes these enhancements into a texture, then using WebGL, composites this texture on top of the original video image to create the final improved frame.

It is essential that the timestamp of the metadata exactly matches the timestamp of the video image, which can be achieved using JavaScript, as they are provided in different ways by the browser.

Each video player is coded differently. With some (such as hls.js) it may be possible just to hotwire global JavaScript properties and functions, and provide the integration through a patch script. With others the only option may be to compile from source, and bundle an entire version of the library.

The steps taken in hls.js serve as an example and starting point to adding LCEVC support to the video technology you wish to target. Please refer to the file lcevc_patch_hls.js when reading this as it includes the methods in that player which must be overidden to invoke LCEVC-enhancement decoding and equivalent methods should be identified in the player you are using.

1. Opening and Closing LCEVC

The recommended way to initialise the LCEVC object is through an attachLcevc function that should be called directly after the player has been created.

// Get reference to video that lcevc will use for input.
video = document.getElementById('video');

// Get reference to canvas that lcevc will use for rendering.
canvas = document.getElementById('canvas');

// Set lcevc config options.
const lcevcDilConfig = {
  debugStats: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'top',
  }
};

// Create lcevc instance within hls object.
hls.attachLcevc(canvas, lcevcDilConfig);

A reference is passed to the canvas (where the frames will be rendered in WebGL) along with the configuration options (some of which can not be modified after creation).

The function is defined inside the patch file as an extension of the hls.js prototype. By listening to the MEDIA_ATTACHING and MEDIA_DETACHING events, the player is able to create an instance of the LCEVC object in memory and remove it from memory when the player closes, seamlessly, without the host page needing any extra script.

// attachLcevc connects the new hls player to the lcevc player
Hls.prototype.attachLcevc = function (canvas, config) {
  this.canvas = canvas;
  this.lcevcDilConfig = config;

  // attachMedia
  this.on(Hls.Events.MEDIA_ATTACHING, (event, data) => {
    this.lcevcDil = new LcevcDil.LcevcDil(
      data.media,
      this.canvas,
      this.lcevcDilConfig
    );
  });

  // detachMedia removes the lcevc components from memory
  this.on(Hls.Events.MEDIA_DETACHING, (event) => {
    if(this.lcevc) {
      this.lcevcDil.close();
      this.lcevcDil = null;
    }
  });

  //...
}

2. Appending Buffer Data

The main job of any video stream player is to parse the manifest file (m3u8) and manage the network, so that the correct video byte data is downloaded from the server and attached to the video player at the right time.

Each of the video technologies mentioned; hls.js, Video.js and Shaka contain a function which appends incoming byte data to a MediaSource Buffer object. At this point video data should be intercepted and a copy of the data also passed to the LCEVC decoder.

// Integration lines of code.
if(
  lcevcDil &&
  (segment.type === 'video' || segment.type === 'audiovideo')
) {
  lcevcDil.appendBuffer(
    segment.data.buffer,
    segment.type,
    loadLevel,
  );
}

NOTE: The LCEVC function “lcevc.appendBuffer” passes the arrayBuffer to a WebWorker which demuxes and decodes it, then stores the extracted LCEVC header packets.

3. Maintaining the Profile Level and PTS Drift

An adaptive bitrate video stream can be composed of video segments with different bitrates (to accommodate changing network conditions) that are concatenated to create a seamless playback inside a single video tag. Because these segments might have slightly different start and end timestamps it is possible for the residual metadata to become misaligned with the base video layer.

Every event of the video player that indicates that the PTS offset has changed when parsing a new fragment. Sometimes they come with extra information like type of fragment, level and drift time. This extra data helps LCEVC with the synchronisation of the residuals.

In hls.js the following event is used:

// Add level PTS update event to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_PTS_UPDATED, function levelPTSUpdated(event, data) {
  if (hls.lcevcDil) {
     hls.lcevcDil.newPts(data.start, data.end, data.type, data.level, data.drift);
  }
});

Whenever the player indicates that the network has changed and so the settings of the video profile the store of LCEVC residuals (collected from metadata) must be flushed. This means whenever a level switch occurs, or the coder changes video source.

In hls.js the following events are used:

// loadSource resets the V-Nova LCEVC decoder and frees memory when hls loads
// a new stream.
hls.on(Hls.Events.MANIFEST_LOADING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.resetDecoder();
  }
});

// Buffer flushing events clears residuals when buffer flushed.
hls.on(Hls.Events.BUFFER_FLUSHING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.flushBuffer(data.startOffset, data.endOffset);
  }
});

// Send level switching event data level to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_SWITCHING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.levelSwitchingEvent(data.level);
  }
});

// Send level switched event data level to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_SWITCHED, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.levelLoadedEvent(data.level);
  }
});

API

The interface between the webpage script and the LCEVC-enhanced player is maintained by a simple API using the following properties, methods, and events.

Configurations options

Configurations parameters can be provided to lcevc_dil.min.js to create an lcevcDil object.

var lcevcDilConfig = {
  renderAtDisplaySize: true,
  logLevel: LcevcDil.LogLevel.None,
  dps: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'bottom',
    offsetY: 24,
    offsetX: 0,
    colorPlayed: 0x688dd3,
    colorBuffered: 0x5e7ebd,
    fullscreenElement: document.querySelector('#player')
  }
};

renderAtDisplaySize

Use the display size instead of the video size when rendering. Useful when the display size is lower than the video size. For example, if the video is 1080p but the player size is 720p, it will render at 720p.

Default: True

logLevel

Set the lcevc log level. Depending on the level specified, different levels of message feedback detail will be printed in the console.

Default: LcevcDil.LogLevel.NONE

Dynamic Performance Scaling

Enable or disable dynamic performance scaling. Dynamic performance scaling monitors playback to ensure that the number of frames being dropped due to constrained system resources hasn't reached an unacceptable level. If the decoding system is struggling, LCEVC decoding will be temporarily paused with the video just displaying the base video decode.

Default: True

playerControls

Enables a simple player controller with some customisation options. It is used to debug the actions of a player controller. Also, it give some info about the stream like video size, selected profile, frame rate and timestamp. It can be configured with the following parameters:

  • enabled: True to enable it, otherwise False.

  • position: CSS style position of the controls.

  • offsetY: Vertical position offset of the controls.

  • offsetX: Horizontal position offset of the controls.

  • colorPlayed: Hex color of the played time bar.

  • colorBuffered: Hex color of the buffered video.

  • fullscreenElement: HTMLElement of the player.

Default: null

Version control

LcevcDil.DIL_BUILD_DATE

Returns the build date.

Log level API

LcevcDil.LogLevel

Log level enumerate:

  • NONE: 0.

  • ERROR: 1.

  • WARNING: 2.

  • INFO: 3.

  • DEBUG: 4.

  • VERBOSE: 5.

LcevcDil API

LcevcDil(video, canvas, configOptions)

Construct a new lcevc object using the HTMLVideoElement of the video, HTMLCanvasElement of the display canvas and the lcevc configuration options.

LcevcDil.video get

Return the HTMLVideoElement used for the video.

LcevcDil.isFullscreen get

Return True if the video is fullscreen, otherwise False.

LcevcDil.isLive get

Return True if the video is live, otherwise False.

LcevcDil.isPerformanceScalingEnabled get

Return True if dynamic performance scaling is enabled, otherwise False.

LcevcDil.enablePerformanceScaling(value)

Enable or disable the dynamic performance scaling.

LcevcDil.isPerformanceScalingActive get

Return True if the performance scaling is active, otherwise False.

LcevcDil.isLcevcEnabled get

Return True if lcevc data is enabled, otherwise False.

LcevcDil.lcevcDataDetected get

Return True if lcevc data is detected in the video, otherwise False.

LcevcDil.frameWidth get

Return the width size of the displayed frame.

LcevcDil.frameHeight get

Return the height size of the displayed frame.

LcevcDil.currentLevel get

Return the number value of the current level.

LcevcDil.firstLcevcSegmentLoaded get

Return True if a segment with lcevc has been loaded, otherwise False.

LcevcDil.aspectRatio get

Returns the value of the aspect ratio of the video.

LcevcDil.frameRate get

Returns the frame rate of the video.

LcevcDil.getConfigOption(option)

Get the value of the option of the lcevc configuration.

LcevcDil.setConfigOption(option, value)

Set the option with the value to the lcevc configuration.

LcevcDil.logLevel get/set

Get or set the log level to the given Lcevc.LogLevel (default NONE).

LcevcDil.profileIn get

Get the input color profile.

LcevcDil.setProfileIn(profileName)

Set the input color profile to the given profile name.

LcevcDil.profileOut get

Get the output color profile.

LcevcDil.setProfileOut(profileName)

Set the output color profile to the given profile name.

LcevcDil.onFullscreen(enable)

Tell LcevcDil that the player has trigger a fullscreen event.

LcevcDil.clearTemporal()

Clears the lcevc temporal buffer. This will remove the current residuals and should only be used when moving or seeking in the video or when changing the profile.

LcevcDil.setCurrentLevel(level)

Notifies lcevc that a level loaded event has been triggered passing the new level. This ensures that LCEVC data is managed correctly upon loading a new level.

LcevcDil.setLevelSwitching(level)

Notifies lcevc that a level switching event has been triggered passing the new level. This ensures that LCEVC data is managed correctly on the transition between levels.

LcevcDil.appendBuffer(data, type, level, start, end)

Integration function only.

Receives video buffer data and sends it to the worker to detect and extract lcevc data. The arguments are:

  • data: The video buffer data.

  • type: The type of the data, can be 'video', 'audio' or 'audiovideo'.

  • level: The current ABR profile/rendition/level ID of the data.

  • start (optional): The start time of the buffer.

  • end (optional): The end time of the buffer.

LcevcDil.flushBuffer(startTime, endTime)

Flush the residual and drift data of the given interval of time in milliseconds.

LcevcDil.resetBuffer()

Reset the internal buffers. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

LcevcDil.newPts(start, end, type, level, drift)

Integration function only.

Instructs lcevc that new information has been updated after parsing a fragment.

The type, level and drift are optional, but providing them helps lcevc to maintain correct synchronisation between the base video and decoded enhancement layers.

  • start: The start time in milliseconds of the fragment.

  • end: The end time in milliseconds of the fragment.

  • type: Type of the fragment, can be 'video' or 'audio'.

  • level: The level of the parsed fragment.

  • drift: The drift amount of time.

LcevcDil.displayError(message)

Prints an error message in the canvas where the video is displayed.

LcevcDil.clearError()

Clears the error message of the canvas if any.

LcevcDil.close()

Closes lcevc and releases memory in use. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

Events API

LcevcDil.on(type, listener)

Attach an event listener of Lcevc.Eventstype to lcevc.

LcevcDil.off(type, listener)

Remove the event listener of the given Lcevc.Eventstype.

Runtime events

LcevcDil.Events.PERFORMANCE_DROPPED

Occurs when Dynamic Performance Scaling is enabled, and GPU system resources are heavily constrained. 

lcevcDil.on( hls.lcevc.Events.PERFORMANCE_DROPPED, function(){
    console.log('Performance Dropped');
}

LcevcDil.Events.PERFORMANCE_RESTORED

Occurs after a PERFORMANCE_DROPPED event, when the GPU is able to run at optimal speed again. LCEVC enhancement decoding has been restored.

lcevcDil.on( hls.lcevc.Events.PERFORMANCE_RESTORED, function(){
    console.log('Performance Restored');
}
PreviousDecoder Integration Layer (DIL)NextNDK

Last updated