Introduction

This section provides an overview of the HTML5 playback components included as part of the V-Nova LCEVC SDK and provides instructions on how to integrate them into different HTML5 players. The primary reference integration for V-Nova LCEVC is based on the hls.js player so the majority of examples given use this as a reference for integrating into your player. Further advice on integration LCEVC with other players and the API calls presented by the V-Nova LCEVC components are given after the detailed overview of working with hls.js.

LCEVC-enhanced HTML5 pipeline

As shown in Figure 1, the main change in the video pipeline due to enable LCEVC decoding is to hide the original video tag region and then play back video on a new canvas region. This new canvas region and passing the MSE SourceBuffers into the LcevcDil.js module are the main requirements of the modified video pipeline. The following sections provide further detail on these aspects as well as integration with web based video players.

The HTML5 decoding takes the following inputs:

1. Video segment as an MSE SourceBuffer that is also fed to the HTML Video Element;

2. Video element for the base decoded video picture;

3. A div element as destination for subtitles;

4. Canvas element as destination for the LCEVC-enhanced decoded video.

By using the requestAnimationFrame API, the LCEVC-enhanced decoder extracts each video frame from the video element and renders the LCEVC-enhanced output frame on the given canvas element.

Since subtitles (e.g WebVTT) are typically rendered by the browser on the video tag, there is currently no way for the LCEVC decoder to extract that region and move it to the canvas. As a result, the solution currently offered by the decoder is to render the cues via simple, yet customizable, div rendering on top of the canvas element.

Supported browsers and platforms

The following capabilities are required in order to integrate V-Nova LCEVC HTML5 playback:

• WebGL 1.0 with FrameBuffer capability. This is almost universal among current browsers and systems.

• WebWorkers / WebAssembly.

• WebAudio with Delay node functionality.

Typical steps taken to verify browser type and feature required including example code are described below:

Step 1: Detect the supported browser and browser version.

The supported browser and versions for Windows:

• Chrome (version ≥ 57)

• Firefox (version ≥ 52)

• Edge (version ≥ 16)

The supported browser and versions for MacOS:

• Chrome (version ≥ 57)

• Firefox (version ≥ 52)

• Safari (version ≥ 11)

Sample code:

var nVer = navigator.appVersion;
var nAgt = navigator.userAgent;
var browserName = navigator.appName;
var fullVersion, verOffset, majorVersion;
​
// In Chrome, the true version is after "Chrome".
if ((verOffset = nAgt.indexOf('Chrome')) !== -1)
{
	browserName = 'Chrome';
	fullVersion = nAgt.substring(verOffset + 7);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Safari, the true version is after "Safari" or after "Version".
else if ((verOffset = nAgt.indexOf('Safari')) !== -1)
{
	browserName = 'Safari';
	fullVersion = nAgt.substring(verOffset + 7);
	if ((verOffset = nAgt.indexOf('Version')) !== -1)
		fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Firefox, the true version is after "Firefox".
else if ((verOffset = nAgt.indexOf('Firefox')) != -1)
{
	browserName = 'Firefox';
	fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
//Edge
else if ((verOffset = nAgt.indexOf('Edge')) != -1)
{
	browserName = 'Edge';
	fullVersion = nAgt.substring(verOffset + 5);
	majorVersion = parseInt('' + fullVersion, 10);
}
else
{
	//Browser not supported.
}

Step 2: Detect the OS.

It is advised that the supported browser for the respective OS is used, as specified in Step 1.

var OSName;
if (navigator.appVersion.indexOf('Win') != -1) OSName = 'Windows';
if (navigator.appVersion.indexOf('Mac') != -1) OSName = 'MacOS';

Step 3: Check that WebGL 1.0 is available.

Create a canvas element. Note that the canvas is not added to the document itself, so it is never displayed in the browser window.

var canvas = document.createElement('canvas');
​
// Get WebGLRenderingContext from canvas element.
var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');
​
if(gl && gl instanceof WebGLRenderingContext) {
	// Congratulations! Your browser supports WebGL.
} else {
	// Your browser may not support WebGL.
}

Step 4: Check if the browser supports/does not support WebWorkers.

if (window.Worker) {
	// Congratulations! Your browser supports WebWorkers.”
} else {
	// Your browser does not support WebWorkers.
}

Step 5: If all of the preceding requirements(steps) are satisfied, then proceed to the video playback. Otherwise, display an alert message to the user.

Components

The HTML5 delivery for the LCEVC SDK consists of the following sets of Javascript and Web Assembly files specific to decoding LCEVC-enhanced streams:

• liblcevc_dpi.js: V-Nova LCEVC JavaScript decoder library.

• liblcevc_dpi.wasm: V-Nova LCEVC WebAssembly decoder library.

• lcevc_dil.min.js: V-Nova LCEVC JavaScript player.

• lcevc_dil_patch_hls.js: V-Nova LCEVC JavaScript functions that are used in conjunction with Hls.js

hls.js

The V-Nova LCEVC SDK uses the hls.js* application for testing and verification of sample streams which can also be provided. The section below outlines the steps needed to integrate the LCEVC decoder modifications to a player based on hls.js.

(*) hls.js is a JavaScript library which implements an HTTP Live Streaming client.

How to integrate LCEVC into an existing hls.js player

Step 1: To integrate LCEVC into an existing hls.js player, you need to include all the scripts under the "demo/js/VNova-LCEVC" folder.

Your HTML page should include the following scripts in the header:

<!-- a version of hls.js -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/hls.js/1.0.4/hls.js"></script>
​
<!-- vnova lcevc -->
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil.min.js"></script>
<script type="text/javascript" src="js/VNova-LCEVC/liblcevc_dpi.js"></script>
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil_patch_hls.js"></script>

Notes:

• A version of "hls.js" will need to be included. Currently, version 1.0.4 is supported.

• The file "lcevc_dil_patch_hls.js" should be included last, as this needs the other scripts to be loaded first.

• All the VNova-LCEVC scripts must be present in the same folder, including "liblcevc_dpi.wasm".

Or it also possible to reference these sources hosted in the AWS cloud by V-Nova. These are updated periodically to the latest versions:

<script type="text/javascript" src="https://d3mfda3gpj3dw1.cloudfront.net/LCEVC_HTML/dil.js/liblcevc_dpi.js"></script>
<script type="text/javascript" src="https://d3mfda3gpj3dw1.cloudfront.net/LCEVC_HTML/dil.js/lcevc_dil.min.js"></script>
<script type="text/javascript" src="https://d3mfda3gpj3dw1.cloudfront.net/LCEVC_HTML/dil.js/lcevC_dil_patch_hls.js"></script>

Step 2: In the HTML, the code used to load the hls.js player must be modified slightly from its normal usage, to integrate the LCEVC enhancement decoding capabilities, as follows:

Code Changes

To check if hls.js is supported or not:

if (Hls.isSupported()) {
	if (hls) {
		hls.destroy();
		hls = null;
	}
} else {
	console.log('Playback is not supported');
}

To create a new instantiation of hls.js:

hls = new Hls({
	debug:             false,
	enableWorker:      true,
	defaultAudioCodec: null
});

To initialise LCEVC within the hls player:

var canvas = document.createElement('canvas');
var lcevcDilConfig = {
    // optional settings here
};
​
hls.attachLcevc(canvas, lcevcDilConfig);

The parameters are defined as follows:

canvas

This is an HTMLCanvasElement object and is used to display the rendered frames / output of the video after V-Nova LCEVC has been applied.

Its visibility, and pixel size are both controlled automatically by the player. Please note that the dimensions (in pixels) are set based on the size of the initial video and of the LCEVC-enhanced content being rendered. It is not guaranteed to always be a fixed size and may change dynamically.

The canvas element should mimic the screen size and page position of the video element, via the user’s CSS pixel size, however, this should not be forced, as it is controlled by the player.

The canvas element will be a WebGL canvas, and therefore it is not possible to use any of the HTML5 2D drawing context methods.

lcevcDilConfig

This is an object block containing settings with which V-Nova LCEVC is initialised. It is automatically configured with optimal settings.

Then just attach the video element as normal:

var video = document.getElementById('video');
hls.attachMedia(video);

The video variable is an HTMLVideoElement object and contains the video, exactly as it would for the regular use of hls.js. Once V-Nova LCEVC has been loaded from the header data, this video element is automatically made invisible, using dynamic CSS. A canvas element is then used to display the rendered frames.

Autoplay when loaded:

hls.on(Hls.Events.BUFFER_CREATED, function(event, data) {
	window.setTimeout(function() {
		video.play();
	}, 500);
});

Integrating LCEVC with other HTML5 video technologies

The primary LCEVC reference integration is with hls.js, version 1.0.4. This section outlines the steps and considerations needed to integrate it with other players such as Video.js or Shaka, and how to write your own patch file.

Overview

The general concept is that the base layer of the video (the image of the frame) is provided by the browser's native video tag, while the LCEVC enhancement data is provided by extra data stored in the video stream. The supplied LCEVC library decodes these enhancements into a texture, then using WebGL, composites this texture on top of the original video image to create the final improved frame.

It is essential that the timestamp of the metadata exactly matches the timestamp of the video image, which can be achieved using JavaScript, as they are provided in different ways by the browser.

Each video player is coded differently. With some (such as hls.js) it may be possible just to hotwire global JavaScript properties and functions, and provide the integration through a patch script. With others the only option may be to compile from source, and bundle an entire version of the library.

The steps taken in hls.js serve as an example and starting point to adding LCEVC support to the video technology you wish to target. Please refer to the file lcevc_patch_hls.js when reading this as it includes the methods in that player which must be overidden to invoke LCEVC-enhancement decoding and equivalent methods should be identified in the player you are using.

1. Opening and Closing LCEVC

The recommended way to initialise the LCEVC object is through an attachLcevc function that should be called directly after the hls.js player has been created.

// Create hls object as usual.
hls = new Hls({ debug: false, enableWorker: true, defaultAudioCodec: undefined });
​
// Get reference to canvas that lcevc will use for rendering.
canvas = document.getElementById('canvas');
​
// Set lcevc config options.
const lcevcDilConfig = {
  debugStats: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'top',
  }
};
​
// Create lcevc instance within hls object.
hls.attachLcevc(canvas, lcevcDilConfig);

A reference is passed to the canvas (where the frames will be rendered to in WebGL) along with the configuration options (some of which can not be modified after creation).

The function is defined inside the patch file as an extension of the hls.js prototype. By listening to the MEDIA_ATTACHING and MEDIA_DETACHING events, hls.js is able to create an instance of the LCEVC object in memory and remove it from memory when the hls.js player closes, seamlessly, without the host page needing any extra script.

// attachLcevc connects the new hls player to the lcevc player
Hls.prototype.attachLcevc = function (canvas, config) {
  this.canvas = canvas;
  this.lcevcDilConfig = config;
​
  // attachMedia
  this.on(Hls.Events.MEDIA_ATTACHING, (event, data) => {
    this.lcevcDil = new LcevcDil.LcevcDil(
      data.media,
      this.canvas,
      this.lcevcDilConfig
    );
  });
​
  // detachMedia removes the lcevc components from memory
  this.on(Hls.Events.MEDIA_DETACHING, (event) => {
    if(this.lcevc) {
      this.lcevcDil.close();
      this.lcevcDil = null;
    }
  });
​
  //...
}

2. Appending Buffer Data

The main job of any video stream player is to parse the manifest file (m3u8) and manage the network, so that the correct video byte data is downloaded from the server and attached to the video player at the right time. For LCEVC to work, this data is intercepted and the portion of it containing LCEVC headers is read.

Each of the video technologies mentioned; hls.js, Video.js and Shaka contain a function which appends incoming byte data to a MediaSource Buffer object. At this point video data should be intercepted and a copy of the data passed to LCEVC.

For hls.js there are two ways of doing it depending on the version of hls.js being used. From version 1.0.0 onwards, a new event is called when a new buffer is parsed, so by listening to this event we can get the buffer, the time and the level from it and append it to LCEVC:

if (Hls.version > '0.14.17') {
  this.on(Hls.Events.BUFFER_APPENDING, function bufferFlushing(_, data) {
    if ((data.type === 'video' || data.type === 'audiovideo') && this.dil) {
      this.lcevcDil.appendBuffer(
        data.data,
        data.type,
        data.frag.level,
        data.frag.start,
        data.frag.start + data.frag.duration
      );
    }
  });
}

For 0.14.2 or below, we override the function named “doAppending” from the BufferController object. The original line in the hls.js source code simply adds data to the hls.js source buffer:

// Original line of code.
sb.appendBuffer(segment.data);

which is modified to call an additional LCEVC function appendBuffer inside the LCEVC decoder before adding the data to the source buffer:

// Integration lines of code.
if(
  hls.lcevcDil &&
  (segment.type === 'video' || segment.type === 'audiovideo')
) {
  hls.lcevcDil.appendBuffer(
    segment.data.buffer,
    segment.type,
    hls.loadLevel,
  );
}
// Original line of code.
sb.appendBuffer(segment.data);

NOTE: The LCEVC function “lcevc.appendBuffer” passes the arrayBuffer to a WebWorker which demuxes and decodes it, then stores the extracted LCEVC header packets.

3. Maintaining the Profile Level and PTS Drift

An adaptive bitrate video stream can be composed of video segments with different bitrates (to accommodate changing network conditions) that are concatenated to create a seamless playback inside a single video tag. Because these segments might have slightly different start and end timestamps it is possible for the residual metadata to become misaligned with the base video layer.

Every event of the video player that indicates that the PTS offset has changed when parsing a new fragment. Sometimes they come with extra information like type of fragment, level and drift time. This extra data helps LCEVC with the synchronisation of the residuals.

In hls.js the following event is used:

// Add level PTS update event to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_PTS_UPDATED, function levelPTSUpdated(event, data) {
  if (hls.lcevcDil) {
     hls.lcevcDil.newPts(data.start, data.end, data.type, data.level, data.drift);
  }
});
​

Whenever the player indicates that the network has changed and so the settings of the video profile the store of LCEVC residuals (collected from metadata) must be flushed. This means whenever a level switch occurs, or the coder changes video source.

In hls.js the following events are used:

// loadSource resets the V-Nova LCEVC decoder and frees memory when hls loads
// a new stream.
hls.on(Hls.Events.MANIFEST_LOADING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.resetDecoder();
  }
});
​
// Buffer flushing events clears residuals when buffer flushed.
hls.on(Hls.Events.BUFFER_FLUSHING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.flushBuffer(data.startOffset, data.endOffset);
  }
});
​
// Send level switching event data level to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_SWITCHING, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.levelSwitchingEvent(data.level);
  }
});
​
// Send level switched event data level to ensure the correct functionality
// of V-Nova LCEVC.
hls.on(Hls.Events.LEVEL_SWITCHED, (event, data) => {
  if(hls.lcevcDil) {
    hls.lcevcDil.levelLoadedEvent(data.level);
  }
});

API

The interface between the webpage script and the LCEVC-enhanced player is maintained by a simple API using the following properties, methods, and events.

Configurations options

Configurations parameters can be provided to lcevc_dil.min.js to create an lcevcDil object.

var lcevcDilConfig = {
  renderAtDisplaySize: true,
  logLevel: LcevcDil.LogLevel.None,
  dps: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'bottom',
    offsetY: 24,
    offsetX: 0,
    colorPlayed: 0x688dd3,
    colorBuffered: 0x5e7ebd,
    fullscreenElement: document.querySelector('#player')
  }
};

renderAtDisplaySize

Use the display size instead of the video size when rendering. Useful when the display size is lower than the video size. For example, if the video is 1080p but the player size is 720p, it will render at 720p.

Default: True

logLevel

Set the lcevc log level. Depending on the level specified, different levels of message feedback detail will be printed in the console.

Default: LcevcDil.LogLevel.NONE

Dynamic Performance Scaling

Enable or disable dynamic performance scaling. Dynamic performance scaling monitors playback to ensure that the number of frames being dropped due to constrained system resources hasn't reached an unacceptable level. If the decoding system is struggling, LCEVC decoding will be temporarily paused with the video just displaying the base video decode.

Default: True

playerControls

Enables and a simple player controller with some customisation options. It is used to debug the actions of a player controller. Also, it give some info about the stream like video size, selected profile, frame rate and timestamp. It can be configured with the following parameters:

• enabled: True to enable it, otherwise False.

• position: CSS style position of the controls.

• offsetY: Vertical position offset of the controls.

• offsetX: Horizontal position offset of the controls.

• colorPlayed: Hex color of the played time bar.

• colorBuffered: Hex color of the buffered video.

• fullscreenElement: HTMLElement of the player.

Default: null

Version control

LcevcDil.DIL_BUILD_DATE

Returns the build date.

Log level API

LcevcDil.LogLevel

Log level enumerate:

• NONE: 0.

• ERROR: 1.

• WARNING: 2.

• INFO: 3.

• DEBUG: 4.

• VERBOSE: 5.

LcevcDil API

LcevcDil(video, canvas, configOptions)

Construct a new lcevc object using the HTMLVideoElement of the video, HTMLCanvasElement of the display canvas and the lcevc configuration options.

LcevcDil.video get

Return the HTMLVideoElement used for the video.

LcevcDil.isFullscreen get

Return True if the video is fullscreen, otherwise False.

LcevcDil.isLive get

Return True if the video is live, otherwise False.

LcevcDil.isPerformanceScalingEnabled get

Return True if dynamic performance scaling is enabled, otherwise False.

LcevcDil.enablePerformanceScaling(value)

Enable or disable the dynamic performance scaling.

LcevcDil.isPerformanceScalingActive get

Return True if the performance scaling is active, otherwise False.

LcevcDil.isLcevcEnabled get

Return True if lcevc data is enabled, otherwise False.

LcevcDil.lcevcDataDetected get

Return True if lcevc data is detected in the video, otherwise False.

LcevcDil.frameWidth get

Return the width size of the displayed frame.

LcevcDil.frameHeight get

Return the height size of the displayed frame.

LcevcDil.currentLevel get

Return the number value of the current level.

LcevcDil.firstLcevcSegmentLoaded get

Return True if a segment with lcevc has been loaded, otherwise False.

LcevcDil.aspectRatio get

Returns the value of the aspect ratio of the video.

LcevcDil.frameRate get

Returns the frame rate of the video.

LcevcDil.getConfigOption(option)

Get the value of the option of the lcevc configuration.

LcevcDil.setConfigOption(option, value)

Set the option with the value to the lcevc configuration.

LcevcDil.logLevel get/set

Get or set the log level to the given Lcevc.LogLevel (default NONE).

LcevcDil.profileIn get

Get the input color profile.

LcevcDil.setProfileIn(profileName)

Set the input color profile to the given profile name.

LcevcDil.profileOut get

Get the output color profile.

LcevcDil.setProfileOut(profileName)

Set the output color profile to the given profile name.

LcevcDil.onFullscreen(enable)

Tell LcevcDil that the player has trigger a fullscreen event.

LcevcDil.clearTemporal()

Clears the lcevc temporal buffer. This will remove the current residuals and should only be used when moving or seeking in the video or when changing the profile.

LcevcDil.setCurrentLevel(level)

Notifies lcevc that a level loaded event has been triggered passing the new level. This ensures that LCEVC data is managed correctly upon loading a new level.

LcevcDil.setLevelSwitching(level)

Notifies lcevc that a level switching event has been triggered passing the new level. This ensures that LCEVC data is managed correctly on the transition between levels.

LcevcDil.appendBuffer(data, type, level, start, end)

Integration function only.

Receives video buffer data and sends it to the worker to detect and extract lcevc data. The arguments are:

• data: The video buffer data.

• type: The type of the data, can be 'video', 'audio' or 'audiovideo'.

• level: The current level of the data.

• start (optional): The start time of the buffer.

• end (optional): The end time of the buffer.

LcevcDil.flushBuffer(startTime, endTime)

Flush the residual and drift data of the given interval of time in milliseconds.

LcevcDil.resetBuffer()

Reset the internal buffers. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

LcevcDil.newPts(start, end, type, level, drift)

Integration function only.

Instructs lcevc that new information has been updated after parsing a fragment.

The type, level and drift are optional, but providing them helps lcevc to maintain correct synchronisation between the base video and decoded enhancement layers.

• start: The start time in milliseconds of the fragment.

• end: The end time in milliseconds of the fragment.

• type: Type of the fragment, can be 'video' or 'audio'.

• level: The level of the parsed fragment.

• drift: The drift amount of time.

LcevcDil.displayError(message)

Prints an error message in the canvas where the video is displayed.

LcevcDil.clearError()

Clears the error message of the canvas if any.

LcevcDil.close()

Closes lcevc and releases memory in use. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

Events API

LcevcDil.on(type, listener)

Attach an event listener of Lcevc.Eventstype to lcevc.

LcevcDil.off(type, listener)

Remove the event listener of the given Lcevc.Eventstype.

Runtime events

LcevcDil.Events.PERFORMANCE_DROPPED

Occurs when Dynamic Performance Scaling is enabled, and GPU system resources are heavily constrained.

lcevcDil.on( hls.lcevc.Events.PERFORMANCE_DROPPED, function(){
    console.log('Performance Dropped');
}

LcevcDil.Events.PERFORMANCE_RESTORED

Occurs after a PERFORMANCE_DROPPED event, when the GPU is able to run at optimal speed again. LCEVC enhancement decoding has been restored.

lcevcDil.on( hls.lcevc.Events.PERFORMANCE_RESTORED, function(){
    console.log('Performance Restored');
}