> This is part 1 of 2 of the full documentation (pages 1–100 of 104).
> The content is paginated: fetch every part to see all of it.
> Next part: https://docs.v-nova.com/technologies/llms-full.txt/1
> Page index: https://docs.v-nova.com/technologies/llms.txt

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.v-nova.com/technologies/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

---

# Home

Welcome to the V-Nova Documentation Site!

Learn how to integrate V-Nova's compression technologies and PresenZ format into your apps:

* **V-Nova LCEVC**: Comprehensive documentation, including both encoder and decoder for SDK and NDK, along with encoder and decoder integrations. Resources cover reference implementations such as ExoPlayer & VLCkit, as well as applications like FFmpeg with LCEVC, providing flexible options to incorporate MPEG-5 LCEVC enhancements across various levels of your software stack.
* **SMPTE VC-6**: Implementation guides for our state-of-the-art VC-6 compression technology, designed for high-quality imaging and video applications.
* **V-Nova PresenZ**: This section provides documentation on V-Nova PresenZ, our volumetric video format, and guidance on integrating it into workflows for innovative VR and AR experiences.

If you’re evaluating LCEVC or VC-6 with your own content, this site provides a full walkthrough to help you achieve optimal results.

Our documentation is continuously updated to reflect the latest versions of our SDKs, integrations, and solutions.

We welcome your feedback — please reach out via the Download Portal or contact us at [**support@v-nova.com**](mailto:support@v-nova.com) with suggestions for improvements.

***

{% content-ref url="/spaces/II3HaKNLFEqGvuN59zwF" %}
[MPEG-5 LCEVC](https://docs.v-nova.com/technologies/mpeg-5.lcevc/)
{% endcontent-ref %}

{% content-ref url="/spaces/g9PKlUvkbpCzUNAxkswo" %}
[SMPTE VC-6](https://docs.v-nova.com/technologies/smpte.vc-6/)
{% endcontent-ref %}

{% content-ref url="/spaces/1djoRwqtVhAs0LOpvzC9" %}
[V-Nova PresenZ](https://docs.v-nova.com/technologies/v-nova.presenz/)
{% endcontent-ref %}


# Overview

## Introduction

V-Nova’s implementation of MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) has been designed for easy integration with existing encoders and decoders, via a software update.

LCEVC adopts a multi-layer approach where any base codec (e.g. h.264, HEVC, AV1 and others), is enhanced via an additional low bitrate stream. LCEVC’s data stream structure is defined by two component streams:

* A base stream decodable by a hardware decoder; and,
* An enhancement stream consisting of one or two enhancement layers suitable for software processing implementation with sustainable power consumption. The enhancement provides improved compression efficiency to existing codecs, and reduces encoding and decoding complexity, for on demand and live streaming applications.

![Sketch Low Complexity Enhancement Video Coding](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/tv2uDLef3o9m5OOsfds1/1.png)

LCEVC brings a new approach to video compression. It leverages a base video codec such as AVC, HEVC, VP9, EVC, AV1 etc. and employs a highly efficient and low complexity enhancement that adds up to two layers of encoded residuals that correct artefacts produced by the base video codec and add detail and sharpness to reconstruct the final output video. By doing so, it improves video compression efficiency while reducing the overall encoding complexity.

![How LCEVC works: Two sub-layers of residual data](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/8MCH2hQDaNRh0mQdVvUr/2.png)

LCEVC can be rapidly implemented in existing decoders with a software update and is inherently backwards-compatible since devices that haven't yet been updated to decode LCEVC happily play the underlying base codec, which further simplifies deployment.

Decoding is extremely light weight, often freeing up resources and matching or reducing battery power consumption vs. native base codec decoding. Native base codec decoding can leverage the existing hardware for that codec, and LCEVC leverages available hardware acceleration for graphics and scaling.&#x20;

### LCEVC Benefits include:

* Reduce Churn – Better video quality and increased resolutions at the same bitrates drive up QoE and customer satisfaction, increasing viewing times for improved ad-monetization and significantly reducing subscriber churn
* Increase Reach – LCEVC enables stable video delivery at lower bitrates (as low as 100kbps) meaning viewers can be reached over the most limited networks.
* Reduced Delivery Costs – Storage and CDN delivery costs can be reduced by up to 50%
* Better Encoding Density – Applying LCEVC to any codec reduces its complexity and can typically improve encoding density by up to 30%
* Compatibility – LCEVC is delivered in industry-standard formats like HLS and MPEG-DASH meaning it is transparent to existing infrastructure (CDNs, DRM, Ad-insertion etc.) and even plays back on players that are not LCEVC-enabled
* Easy To Deploy – Supported by a broad range of encoding and player vendors or easily update systems with SDKs
* Broad Device Support – LCEVC has the same compatibility as the underlying codec it is enhancing.

## V-Nova LCEVC SDK Overview

The V-Nova LCEVC SDK consists of multiple encoding and decoding libraries to facilitate the rapid deployment of LCEVC across all platforms. There are also reference pre-integrations within popular video processing workflows such as FFmpeg and media players across platforms such as iOS, Android and Windows as well as scripted decoding options for HTML5 capable browsers enabling integration into existing video workflows.

The V-Nova LCEVC SDK provides:

* The industry’s first optimized software library for encoding and decoding MPEG-5 LCEVC enhanced streams
* The ability to deploy LCEVC at any level of your software stack with a range of options from low-level APIs to reference implementations in commonly used open-source encoders and players

## SDK Package

The V-Nova LCEVC SDK 3.0 includes the following libraries and reference integrations:

**Core Encoding and Decoding**

* V-Nova LCEVC encoding libraries (EIL)
* V-Nova LCEVC decoding libraries (DIL)
* EIL Header file (optional)
* DIL Header file (optional)
* Product documentation

**Base codecs Plug-in**

LCEVC SDK support the enhancement of multiple codec implementation through a simple plug-in system. Available plug-ins include:

* x264
* x265
* vp8/vp9 (libvpx)
* AV1 (libaom)
* SVT-AV1
* NVENC&#x20;
* QSV
* Xilinx
* MainConcept
* Netint
* ...

and others can be developed and added even after integration within a system.

**Reference Integrations**

LCEVC-enabled builds and patches for:

* FFmpeg/FFplay
* ExoPlayer
* HTML5
* VLCKit
* Windows (MFT)

## LCEVC Encoding & Decoding

Figure 1 below illustrates how LCEVC operates on both encoding and decoding pipelines. The base encoding (whether H.264, HEVC or others) is performed on a down-scaled input at a lower resolution, typically a quarter of the desired output resolution. LCEVC enhancement data is calculated at the two resolutions providing two levels of correction and enhancement. The LCEVC encoder generates the enhancement stream from two inputs: the base encoding and the original uncompressed full resolution video, effectively correcting the quality gap between the two. The LCEVC data can be packaged together with the base elementary stream (for example as Supplemental Enhancement Information (SEI) of the Network Abstraction Layer, NAL), as frame metadata in a WebM container or in an additional data Packet Identifier (PID) in a MPEG-2 TS stream.&#x20;

![ Figure 1 – Logical block diagram of LCEVC-enhanced encoding and decoding](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/LPKQR7GIikUo2txyE7SW/LCEVC-encode-decode-1-1030x414.png)

The LCEVC decoder works at an individual video frame level. As input it takes the decoded low-resolution picture from the base (H.264 or other) video decoder, which is typically provided by a hardware decoder on the device, and the LCEVC enhancement decoded in software to produce a full-resolution picture ready for rendering on the display view.

The V-Nova LCEVC Decoder Integration Layer (DIL) is a software library that extracts, decodes and applies LCEVC on top of the base decoded video during playback. Once integrated into the decoding device, the DIL enables decoding and playback of any LCEVC-enhanced video stream.


# Best practices

This page gives advice on how to get the very best results when applying LCEVC-enhancement to your encoding workflows.

## Introduction

V-Nova LCEVC is a comprehensive SDK for enhancing any encoding and decoding workflow with MPEG-5 Part 2 Low Complexity Enhancement Video Coding.&#x20;

LCEVC provides a boost to the compression efficiency of any existing or future video codec, to achieve higher quality at up to 40% lower bitrates, while accelerating encoding and reducing compute costs by up to 4x.

![LCECV: Less Bitrate Less Processing Power](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/rB7p4H5DBlop7SSmB8wa/LCEVC_bitrate_comp_power.png)

## How to select the optimal bitrates and resolutions for encoding with LCEVC

Our Video Quality Framework allows us to define the ideal LCEVC-enhanced ABR ladder for the specific business requirements of a video service provider.&#x20;

A Convex Hull approach that combines VMAF and subjective video quality assessment has been used to determine the best combination of bitrate/resolution for specific content and base codec.&#x20;

The charts below provides a guideline, based on V-Nova's experience across a broad range of content.

![Native H.264 vs LCEVC H.264](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/LCYGEmuqHvZATVqb4Adf/LCEVC_H.264_COMPARISON.jpg)

![Native H.265 vs LCEVC H.265](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/EZYcCDuxwhrEXFu7toRV/LCEVC_H.265_COMPARISON.jpg)

## Try our LCEVC-enhanced ABR Ladder Builder

The extensive testing and analysis above has been incorporated into easy-to-use excel sheets that provide initial suggestions for how to select optimal LCEVC-enhanced adaptive streaming profiles for your use case. Click on the links below to try them.

* ### [LCEVC-enhanced H.264/AVC](https://docs-vnova-com.s3.eu-west-1.amazonaws.com/LCEVC+%26+H.264+ABR+Ladder+Builder+\(web\).xlsx)
* ### [LCEVC-enhanced H.265/HEVC](https://docs-vnova-com.s3.eu-west-1.amazonaws.com/LCEVC+%26+H.265+ABR+Ladder+Builder+\(web\).xlsx)

## An example LCEVC-enhanced ABR ladder

The table below provides an example, from a report written by industry analyst Jan Ozer.&#x20;

The H.264 ABR Ladder profiles, encoded using x264 (medium preset), are in line what recommended by Apple in the HLS Authoring Specification.&#x20;

The LCEVC H.264 ABR Ladder profiles have been opt**i**mised to achieve both a quality uplift and a reduction in the total average bitrate streamed.

The amount of motion in the source can also determine encoding settings.&#x20;

![Example of H.264 ABR ladder vs LCEVC H.264](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/NnSTWPp0Bvo0P6vz3ptD/image.png)

With LCEVC the ABR Ladder profiles have been reduced from 7 to 6, reducing the total bitrate streamed by 27% (whilst achieving a better quality).&#x20;

Jan Ozer's report confirms that **LCEVC generates both bandwidth** **savings** - between 19% to 22% reduction in the total average bitrate streamed, based on typical playback frequency of profiles -  **and improvement in overall quality of experience**, measures both subjectively via Mean Opinion Score (MOS) and with VMAF.

Full details and results are available [here](https://streaminglearningcenter.com/codecs/lcevc-excels-in-full-ladder-live-use-case-testing.html).&#x20;


# LCEVC test streams

Description of various MPEG-5 LCEVC test streams available from V-Nova and other vendors, to validate a variety of integrations for different applications.

## V-Nova test streams

As the lead developer of MPEG-5 LCEVC, V-Nova provides sample LCEVC streams to support customer evaluation, demonstrations and integration work. These are provided as-is and not as formal compliance streams set. They are intended to help customers get started and to support practical testing of playback or integration scenarios.

V-Nova test streams are divided into:

1. **LCEVC test streams**
   1. Contains streams combined with various base video codec such as AVC, HEVC, VVC combined with the LCEVC-enhancement carried in SEI or a separate track (in a single or separate streams) in practical carriage formats such as MP4 and Transport Stream, with DASH/HLS-style delivery variants &#x20;
   2. Target users: media player vendors, app developers and platform teams
2. **DTV+ test streams**
   1. Contains streams with focus on the TV3.0 (DTV+) use case streams with VVC as base codec
   2. Target users: encoders, SOCs and media players working on Brazilian TV 3.0 (DTV+), TV2.5 terrestrial as well as broadcasters conducting receiver or device-related evaluations
3. **DVP test streams**
   1. Contains streams specifically designed for SOC driver-level integration of LCEVC using the DVP. DVP is a library for LCEVC Hybrid Driver-level Decoder product integration described here: <https://docs.v-nova.com/technologies/mpeg-5.lcevc/decoder/ndk/lcevc-hybrid-driver-level-decoder>
   2. Target users: TV and STB SoC vendors integrating the DVP decoder solution.

Please note that V-Nova sample streams are provided as-is and are not a formal compliance test suite.

To request V-Nova Sample Streams please reach out [**support@v-nova.com**](mailto:support@v-nova.com) our team will manage the streams package access through V-Nova Download Portal <https://download.v-nova.com/>.

## Allegro compliance streams

Allegro DVT, a global leader in digital video compression solutions development, provides the following packages for elementary and system-level compliance:

1. **Sirius elementary streams: MPEG-5 / LCEVC**
   1. The test suite contains:
      1. **Syntax streams**, to ensure that the decoders support all LCEVC enhancement layer coding tools
      2. **Performance streams**, to test decoding speed (i.e. capability to decode worst case streams in real-time)
      3. **Error resilience streams**, to test decoders behaviour with non-compliant streams.
   2. Target users: IP and IC vendors, system vendors (set-top boxes, digital TVs, smartphones, tablets, etc.).
2. **Streamwise system stream: TV3.0 (DTV+)**
   1. Consists of carefully prepared video and audio test streams for self-testing against the relevant parts of the basic broadcast TV 3.0 (DTV+) specification.
   2. Target users: IC vendors, system vendors (set-top boxes, digital TVs, smartphones, tablets, etc.) and broadcasters.

More information on how to purchase can be found from Allegro:

<https://www.allegrodvt.com/products-cs/elementary-streams-mpeg-5-lcevc/>

<https://www.allegrodvt.com/product-streamwise/streamwise-system-streams-tv3-0/><br>

## MPEG/ISO LCEVC elementary streams

MPEG/ISO provides a set of LCEVC elementary bitstreams designed to verify whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-2. These streams are provided as enhancement-layer-only streams with the base codec provided as YUV file.

These streams can be useful for early-stage experimentation or initial validation, particularly for teams that want access to basic LCEVC elementary streams. Please be aware these are a limited sample set rather than a comprehensive compliance suite provided by other commercial vendor.

\
The streams can be purchased from ISO here: <https://www.iso.org/standard/80902.html#amendment>


# Getting started

To evaluate the compression-efficiency and performance benefits of LCEVC-enhanced encoding with your content there are two primary approaches directly available from V-Nova in addition to using encoders from 3rd parties:&#x20;

* Use the [V-Nova Platform](/technologies/v-nova.platform)
* FFmpeg (See below for quick-start instructions)

## FFmpeg Quick Start

This short guide aims to help you through the simple process of installing the LCEVC enabled FFmpeg and then to perform your first encode.

The LCEVC-enabled FFmpeg can be installed under Linux or Windows. The instructions here are specific to Windows. If running on a Linux environment, please note the small differences as described at the [bottom of this page](#7.-using-linux).

### 1.    **Putting the software together**

The LCEVC-enabled FFmpeg build can be easily assembled. You will have received software in the following packages:

* FFmpeg:           the FFmpeg binaries with support for x264 and x265
* lcevc:              the LCEVC libraries both encoder and decoder libraries
* base codecs:  any additional base codecs requested (if applicable)

To set it all up, simply:

1. UnZip the FFmpeg binaries for your operating system into a local directory of your choice (or create a new one). This will be your “FFmpeg directory”.
2. Copy the LCEVC encoder and decoder libraries with their subfolders onto the root  FFmpeg directory
3. Copy any additional base codecs and base codec plug-ins that you may have received from their folder onto the root FFmpeg directory (if applicable).
4. If running in a docker you may need to install extra dependencies. See the '[Putting the software together'](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#putting-the-software-together) section in the main documentation guide for more details.

### **2.     Encode a stream using LCEVC h264**

This release of FFmpeg supports LCEVC with either an x264 or x265 base codec. Additional supported base codecs are available on request from V-Nova.

Most standard FFmpeg command-line options are included, as well as additional options for configuring V-Nova LCEVC.

The following help commands in FFmpeg describe the behaviour of lcevc\_\<codec>:

`.\ffmpeg.exe -help encoder=lcevc_h264`\
`.\ffmpeg.exe -help encoder=lcevc_hevc`

\
An extra command line parameter "eil\_params" is used to control the operation of both the enhancement and base codecs. eil\_params is a semicolon-separated string of parameters and values that are passed to the LCEVC enhancement layer:

`.\ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "<param1>=<value1>;<param2>=<value2>;…" output.ts`

Further details are provided in the following page: [FFmpeg with LCEVC](/technologies/mpeg-5.lcevc/integrations/ffmpeg)

Certain parameters (e.g., bitrate, GOP length and framerate) are needed by both LCEVC and the base codec it enhances. These parameters are ***not*** set through the eil\_params string, but use the standard FFmpeg command line options:

`-b:v`  for video bitrate\
`-g`    for GOP length (suggested default at 2 seconds, frame rate x2)\
`-r`       for framerate

These values are used to calculate initialisation values for both LCEVC and the base codec.

Suggested default command line for LCEVC x264:

`.\ffmpeg.exe -i ${INPUT_PATH} -g 50 -b:v 2100k -c:v lcevc_h264 -base_encoder x264 -r 25 -s 1920x1080 -eil_params "preset=medium;lcevc_tune=vq" ${OUTPUT_NAME}.ts`

&#x20;Suggested default command line for native libx264 if you are conducting comparison tests:

`.\ffmpeg.exe -i ${INPUT_PATH} -g 50 -b:v 3100k -c:v libx264 -preset medium   -bufsize 6200k  -maxrate 3100k -r 25 -s 1920x1080 ${OUTPUT_NAME}.ts`

If you are unsure or require any assistance for a ffmpeg command line conversion, please reach out to V-Nova for support.

&#x20;

### **3.     Encode with test LCEVC enhancement layer**

The following ***test*** can be performed to ensure the decoder / decoded file is correctly working with LCEVC.&#x20;

N.B. The command should be entered on one long line. It is shown here as multiple lines for clarity, but must be typed on a single line.&#x20;

\
`.\ffmpeg.exe -i ${INPUT_PATH} -g 50 -b:v 2100k -c:v lcevc_h264 -base_encoder x264 -r 25 -s 1920x1080 -eil_params "preset=medium;lcevc_tune=vq;encoding_debug_residuals=1"  ${OUTPUT_NAME}.ts`

&#x20;

This test will add moving squares to the LCEVC layer only. Non LCEVC players or decoders will not see these blocks, but a correctly configured LCEVC player will show images like the one below.

N.B. This command must not be used for quality testing. It is simply a tool for functional encoding/ decoding tests.

![Stream with test LCEVC enhancement layer](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/1GXUdioHT8fhUggLeOSs/LCEVC_bouncing_squares.PNG)

### **4.     Decoding with FFmpeg**

For decoding with FFmpeg for either simple playback or to create uncompressed files to use in video-quality metrics testing, please see the [FFmpeg section in the following "Decoding" page](/technologies/mpeg-5.lcevc/decoder/getting-started).

### **5.     Further information**

Further information and more in-depth documentation is available at:

{% content-ref url="/pages/-M53dFBRI-u6uo-IIkgm" %}
[FFmpeg](/technologies/mpeg-5.lcevc/integrations/ffmpeg)
{% endcontent-ref %}

{% content-ref url="/pages/-MSgvvBIembmKwYm9YUp" %}
[Best practices](/technologies/mpeg-5.lcevc/v-nova-lcevc/lcevc-best-practices)
{% endcontent-ref %}

### &#x36;**.     Using Linux**

The commands in this guide are for a windows environment. If running on a Linux environment there are three key differences.

1. All instances of ‘\’ should be swapped with ‘/’
2. The binaries must be made executable by using chmod +x ./ffmpeg
3. A library path must be defined for ffmpeg and ffplay, such as the following: `LD_LIBRARY_PATH=. ./ffmpeg -i ${INPUT_PATH}…`


# SDK


# Encoder Integration Layer (EIL)

## Introduction

This article serves as a reference to the V-Nova LCEVC Encoder Integration Layer and should be read in conjunction with the Encoder Integration Layer (EIL) header, where each function is well documented as to its behaviour, arguments and return conditions. This document outlines the design, flow of functionality, and the properties exposed through the Integration Layers configuration system.

The following figure illustrates the EIL software stack.

![V-Nova LCEVC Encoding Software Stack](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/g29YBy2zx5ypEstzPHJb/V-Nova_LCEVC_Encoder_and_Decoder_v6.png)

The EIL combines V-Nova LCEVC with the base codec, and wrappers for supported base codecs. V-Nova provides example source code below to show how to use the Integration Layer.

### Single-pass versus multi-pass

V-Nova LCEVC is a single-pass encoder and provides the same results as if it were included in a multi-pass implementation, as the codec always operates in a single pass.

Putting it all together, the following scripts can be tested as starting points for CBR and CRF. Assuming that the scripts would be used to encode short test sequences, we included commands to set the rate control window type to “rolling window” mode.

## Functionality

The EIL provides a simple yet powerful API, and is similar in usage to working with other video encoding libraries.

The steps for encoding streams using the V-Nova LCEVC EIL are as follows:

1. Initialise the Integration Layer
2. While there are frames to encode:

   &#x20;   a. Obtain an input frame and pass ownership to the encoder.

   &#x20;   b. Wait for the encoder to complete, then fetch the encoded frame.

   &#x20;   c. Release ownership of the output frame back to the encoder.
3. Shut down the Integration Layer


# EIL Integration Process

## Overview

The integration of the EIL can be summarised in several steps:

1. Creation of the EIL
2. Initialisation of the EIL
3. Passing video frames to the EIL
4. Fetching the encoded bitstream from the EIL&#x20;
5. Flushing the EIL
6. Destruction of the EIL

The above steps, as well as the various options available to integrations, are described below.

### &#x20;**1. Creation of the EIL**

An instance of the EIL is created using `EIL_Open`. An `EILOpenSettings` object is required, which at minimum, must specify the name of a base encoder. This instance of the EIL should be passed to all other functions. The `EILOpenSettings` can be initialised to a sensible default state using `EIL_OpenSettingsDefault`.

The EIL issues internally generated logging messages via a callback. The callback is registered on the `EILOpenSettings` struct. You must provide a callback function pointer before calling Open to receive messages.

```c
void Log(void* userdata, int32_t level, const char* msg)
{
    fprintf(level == EIL_LL_Error ? stderr : stdout, msg);
}

EILContext OpenEIL()
{
    EILOpenSettings settings;
    EILReturnCode   rc = EIL_OpenSettingsDefault(&settings);

    if (rc != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to initialise default open settings: %s\n",
                EIL_GetErrorString(rc));
        return NULL;
    }

    settings.base_encoder = "x264";
    settings.log_callback = Log;
    EILContext context   = NULL;

    if ((rc = EIL_Open(&settings, &context)) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to open the EIL: %s\n", EIL_GetErrorString(rc));
        return NULL;
    }

    return context;
}
```

### 2. Initialisation of the EIL

The EIL is initialised using `EIL_Initialise`. An `EILInitSettings` object is required, in which the encoding properties, such as resolution, framerate and bitrate, are specified. `properties_json` should be set to a json object string with extra optional encoder options.

Upon successful initialisation, the user can then proceed to encoding YUV input.

```c
// context should have been previously created with EIL_Open
bool InitialiseEIL(EILContext context)
{
    EILInitSettings settings;
    EILReturnCode   rc = EIL_InitSettingsDefault(&settings);

    settings.width     = 1920;
    settings.height    = 1080;
    settings.fps_num   = 25;
    settings.fps_denom = 1;
    settings.bitrate   = 3000; // NB: This is in kbps

    // The properties are usually generated from user input
    settings.properties_json = "{ \"encoding_transform_type\": \"dds\" }";

    if ((rc = EIL_Initialise(context, &settings)) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to intiailise the EIL: %s\n", 
                EIL_GetErrorString(rc));
        return false;
    }

    return true;
}
```

### **3. Passing video frames to the EIL**

If the previous steps succeeded, then the EIL is now ready to receive frames.

If the EIL has been configured to not use external input, then a preallocated frame can be fetched from the EIL using `EIL_GetPicture`;  otherwise, an `EILPicture` object should be created at the integration layer and passed to `EIL_Encode`.

To encode YUV frames, perform the following steps:

1. Call `EIL_GetPicture` to obtain an `EILPicture` object for the type of frame to be encoded. The caller now temporarily owns the picture object.
2. Load YUV data into the pre-allocated memory of the plane member of the picture object.
3. Call `EIL_Encode` using the `EILPicture` object to begin encoding the picture. The encoder retakes ownership of the `EILPicture` object, and it is no longer valid to manipulate it. This is a blocking call for the duration of the encoding of a single frame.

```c
bool Encode(EILContext context, int64_t pts)
{
    EILPicture* picture = NULL;
    EILReturnCode rc = EIL_GetPicture(context, EIL_FrameType_Progressive, &picture);

    if (rc != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to obtain picture from the EIL: %s\n",
                EIL_GetErrorString(rc));
        return false;
    }

    // Write the frame data to picture->plane ensuring that the picture->stride
    // for each plane is obeyed.

    picture->pts = pts;

    if ((rc = EIL_Encode(context, picture)) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to encode picture: %s\n", EIL_GetErrorString(rc));
        return false;
    }

    return true;
}

```

It is invalid behaviour to hold onto the `EILOutput` object across multiple frames. The caller must copy and buffer the data if that is the required behaviour.

### **4. Fetching the encoded bitstream from the EIL**&#x20;

After receiving enough frames, the EIL is ready to output encoded data. The encoded data is fetched using the `EIL_GetOutput` function.

* If there is any output data available when `EIL_GetOutput` is invoked, then `EIL_GetOutput` returns `EIL_RC_Success`.
* If there is no data available and no errors occurred, then `EIL_GetOutput` returns `EIL_RC_Finished`.

To fetch the encoded bitstream from the EIL, perform the following steps:

1. The loop call `EIL_GetOutput` returns the following, depending upon the presence of output data:\
   \- If there is output data, then `EILGetOutput` returns `EIL_RC_Success` and populates `EILOutput*` with a valid pointer. The caller now owns the output data object.\
   \- If there is no more output data, then the `EIL_GetOutput` returns `EIL_RC_Finished`, unless an error is generated.\
   **Please be aware this step is important, as the EIL may generate more than one output per single input, for example, when generating an interlaced encode.**
2. Utilise the output data, either by writing it directly out to file or copying it if it must be buffered for output in a muxing system.
3. Call `EIL_ReleaseOutput` to release the output object back to the ownership of the encoder. It is then no longer valid to read from this object.

Once the integration has finished with the output data, it should return it to the EIL using `EIL_ReleaseOutput`. It is perfectly fine to call this method from a separate thread other than `EIL_Encode`.

```c
int HandleOutput(EILContext context)
{
    EILReturnCode rc     = EIL_RC_Success;
    EILOutput*    output = NULL;

    /* Always loop over output. Flush blocks and accumulates may frames worth of
     output. */
    while ((rc = EIL_GetOutput(context, &output)) == EIL_RC_Success)
    {
        // pass the bitstream to a muxer
        mux(output->pts, output->dts, output->data, output->data_length);
		
        /* Immediately release the output. */
        if (EIL_ReleaseOutput(context, output) != EIL_RC_Success)
        {
            fprintf(stderr, "Unable to get output from the EIL: %s\n",
                    EIL_GetErrorString(rc));
            return -1;
        }
    }

    if (rc != EIL_RC_Finished)
    {
        fprintf("Error: Failed to handle output\n");
        return -1;
    }

    return 0;
}
```

### **5. Flushing the EIL**

To signal to the EIL that no more input data is available, simply pass `NULL` to `EIL_Encode`. The EIL then finishes processing any queued video frames. This should only ever be performed at the end of the stream.

To complete encoding, perform the following steps:

1. Call `EIL_Encode` once using a null pointer for the picture object argument, to flush the entire pipeline. This is a blocking call whilst the remaining pictures are encoded. Subsequent calls will have no effect.
2. Repeat step 4 from the encoding process `IL_RC_Finished` is returned.

### **6. Destruction of the EIL and Shutdown**

Once encoding has finished, then the EIL can be destroyed. Call `EIL_Close` to safely shut down and release the encoding context.

```c
void CloseEIL(EILContext* context)
{
    EIL_Close(*context);
    *context = NULL;
}
```


# Error Handling

Almost all of the APIs available from the EIL return `EILReturnCode`. The documentation for each function Error Handling outlines the potential return conditions. Every effort has been made to ensure that invalid usage of the IL is caught and presented with a meaningful return code. Additionally, a log error or warning will typically be issued via the logging mechanism.


# Base Encoder Plugin

Base layer encoding is implemented via a plugin system. Base encoder plugins, along with their corresponding wrappers, are dynamically loaded at runtime, based upon the name of the Base encoder specified in the `EILOpenSettings` structure.&#x20;

The name of the base encoder that the EIL uses for base encoding should not contain the preceding `lcevc_eilp_` (`liblcevc_eilp_` on Linux) or the platform's dynamic library extension. For example, if a plugin is called `liblcevc_eilp_x264.so` or `lcevc_eilp_x264.dll`, then the `base_encoder` string should be `x264`.

The EIL utilises several search paths to locate plugins. This is to assist with different program usage behaviours and EIL integration techniques. The order of search is as follows:

1. The location of the application process.
2. The current working directory.
3. System-defined PATH environment variable.

An API is provided for the EIL to interact with plugins. This allows a user to optionally implement their own base encoder through the plugin system without the need to be embedded within the EIL, and provides the benefits of modularity. Note: This requires the base plugin API and further documentation on how to develop your own base encoder plugin.


# Features

## External Input

The EIL can be configured to accept externally allocated input frames. To do this, the integration should set `external_input` in `EILInitSettings` to true. When configured in this mode, the EIL will not allocate any pictures itself, and so `EIL_GetPicture` will always fail.

In external input mode, the integration must create the `EILPicture` object itself. The object can be default initialised using `EIL_InitPictureDefault`. The integration then must set the `plane` pointers to memory it has allocated and `stride` to the number of bytes per line. The other fields are then used in the same manner as when external input is disabled.

The planes must be valid from the point that the picture is passed to `EIL_Encode`, until the associated output is produced.

The EIL can also accept Android hardware buffers in this mode when the EIL is configured for GPU mode. To do this the picture's `memory_type` is set to `EIL_MT_AHardwareBuffer` and the plane pointers are set to `AHardwareBuffer`.

```c
bool InitialiseExteranlInput(EILContext context)
{
    EILInitSettings settings;
    EILReturnCode   rc = EIL_InitSettingsDefault(&settings);

    settings.width          = 1920;
    settings.height         = 1080;
    settings.fps_num        = 25;
    settings.fps_denom      = 1;
    settings.bitrate        = 3000; // NB: This is in kbps
    settings.external_input = 1;

    // The properties are usually generated from user input
    settings.properties_json = "{ \"encoding_transform_type\": \"dds\" }";

    if ((rc = EIL_Initialise(context, &settings)) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to intiailise the EIL: %s\n",
                EIL_GetErrorString(rc));
        return false;
    }

    return true;
}

bool EncodeExternalInput(EILContext context, int64_t pts)
{
    EILPicture* picture = (EILPicture*)malloc(sizeof(EILPicture));

    if (EIL_InitPictureDefault(picture) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to initialise picture object\n");
        return false;
    }

    // For the example the planes are allocated inline. These could just as
    // easily be set to pointers that already contain the frame data to
    // avoid copying the data.
    // Assuming 1080p YUV420p
    picture->pts        = pts;
    picture->num_planes = 3;
    picture->plane[0]   = malloc(1920 * 1080);
    picture->stride[0]  = 1080;
    picture->plane[1]   = malloc(960 * 540);
    picture->stride[1]  = 540;
    picture->plane[2]   = malloc(960 * 540);
    picture->stride[2]  = 540;
    picture->user_data  = picture;

    // populate the allocated planes with frame data

    EILReturnCode rc = EIL_Encode(context, picture);

    if (rc != EIL_RC_Success)
    {
        for (int i = 0; i < picture->num_planes; ++i)
            free(picture->plane[i]);

        fprintf(stderr, "Unable to encode frame: %s\n", EIL_GetErrorString(rc));
        return false;
    }

    return true;
}

bool HandleOutput(EILContext context)
{
    EILOutput*    output = NULL;
    EILReturnCode rc     = EIL_GetOutput(context, &output);

    if (rc != EIL_RC_Success)
    {
        if (rc != EIL_RC_Finished)
        {
            fprintf(stderr, "Unable to get output from the EIL: %s\n",
                    EIL_GetErrorString(rc));
            return false;
        }

        return true;
    }

    // pass the bitstream to a muxer
    mux(output->pts, output->dts, output->data, output->data_length);

    EILPicture* picture = (EILPicture*)output->user_data;

    for (int i = 0; i < picture->num_planes; ++i)
        free(picture->plane[i]);

    if ((rc = EIL_ReleaseOutput(context, output)) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to release output back to the EIL: %s\n",
                EIL_GetErrorString(rc));
        return false;
    }

    return true;
}
```

## Dynamic Reconfiguration

The EIL is able to reconfigure the target bitrate and framerate, if the base encoder also supports dynamic reconfiguration. This is done by passing `EILReconfigureSettings`, with the new values, to the `EIL_Reconfigure` function.

```c
bool Reconfigure(EILContext context)
{
    EILReconfigureSettings settings;

    if (EIL_ReconfigureSettingsDefault(&settings) != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to default initialise reconfigure settings");
        return false;
    }

    settings.flags     = EIL_RF_Bitrate | EIL_RF_Framerate;
    settings.bitrate   = 5000;
    settings.fps_num   = 30;
    settings.fps_denom = 1;

    EILReturnCode rc = EIL_Reconfigure(context, &settings);

    if (rc != EIL_RC_Success)
    {
        fprintf(stderr, "Unable to reconfigure EIL: %s\n",
                EIL_GetErrorString(rc));
        return false;
    }

    return true;
}
```


# Encoding Configuration Properties

The EIL provides a very powerful configuration system. All configuration properties, except for the basic frame information, are set via a JSON configuration string in the `EILInitSettings` structure.

The configuration system is supported by a comprehensive built-in system of optimised parameter defaults. The EIL employs ideal settings for both the LCEVC encoder and base encoder, based upon key parameters passed to it by the application, e.g., *resolution*, *bitrate*, *tune*.

Properties can be anonymously queried by the `EIL_QueryPropertyGroups` API, and, if the user knows the name of properties, by the `EIL_QueryProperty` API.

Properties specified within the JSON string override any setting provided by the database. Therefore, it is important that an application takes care to specify only properties within the JSON string that the user has explicitly set. For example, in a command-line program, only the options specified by the user in the command line should be included in the JSON string, or, from a GUI, only the options that the user has manually specified a choice for should be included in the JSON string.

The JSON string is processed utilising strong typing correlating to the target property type. If the types are not adhered to, then the EIL will fail to initialise. The user must take care to generate a JSON string that utilises the correct JSON type for each property type. The following table outlines the types and their JSON equivalents:

| EILPropertyType  | JSON type                                            |
| ---------------- | ---------------------------------------------------- |
| EIL\_PT\_Int32   | Number                                               |
| EIL\_PT\_Int64   | Number                                               |
| EIL\_PT\_Uint32  | Number                                               |
| EIL\_PT\_Uint64  | Number                                               |
| EIL\_PT\_Float   | Number                                               |
| EIL\_PT\_Double  | Number                                               |
| EIL\_PT\_String  | String                                               |
| EIL\_PT\_Boolean | Boolean, string (‘0’ = false), or number (0 = false) |
| EIL\_PT\_Blob    | Not used for properties                              |

The JSON string is formatted with a top-level object, and a member per property, as shown in the following example:

```c
{
 "encoding_downsample_luma" : "cubic",
 "temporal_enabled" : true,
 …
}
```

For additional convenience, properties have been assigned to logical groupings, and can optionally contain a description about what the property does. This is so a user may dynamically generate a command-line parser, or a UI, for the publicly available properties


# CBR vs CRF

#### Overview

LCEVC encodes enhanced streams in Constant Bitrate mode (**CBR**) or (either uncapped or capped) Constant Rate Factor (**pCRF**) mode.

#### Rate control window type and length

The LCEVC rate control can work according to different modes: “chunk” (default) or “rolling window”. When in streaming “chunk” mode, the rate controller resets the leaky bucket fill up level at the beginning of each streaming chunk, to avoid unnecessary influence from one chunk to the next (e.g. making a chunk slightly smaller than the target bitrate just because the previous one was slightly bigger, or vice versa). When in “rolling window” mode, instead, the leaky bucket fill up level is never reset.

“Chunk” mode is recommended for ABR chunk-based streaming, while “rolling window” mode is recommended for low-latency video as well as for tests involving short self-similar sequences.

“Chunk” mode is active by default, so there is no need to specify the corresponding setting within `eil_params (rc_pcrf_window_type=chunk).`

To activate the “rolling window” mode, the following command should be used within `eil_params`:

**`rc_pcrf_window_type=rolling`**

By default, the rate control window length is two GOPs, but you can specify a different length in frames with the following command within `eil_params`:

**`rc_pcrf_window_duration_frame=<number of frames in a window>`**

#### CBR

CBR ensures that the same bitrate is maintained throughout the clip, as is required for many streaming video systems. This is achieved in FFmpeg by specifying the target bitrate with `–b:v` parameter, as in the following example (CBR at 2 Mbps).

Note: By default, in FFmpeg, `-b:v` bitrate is interpreted as bps and a suffix for kbps and Mbps must be used, e.g. -b:v 2000k specifies that bitrate = 2 Mbps.

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -b:v 2000k output.ts`

In LCEVC, a CBR stream can have a base following a different rate control paradigm. In LCEVC, the *base encoder* rate control can work with either a CBR base layer (**default**), or a CRF base layer.

* *`rc_pcrf_base_rc_mode=cbr`* (default) is recommended for most codecs. By default, LCEVC will also adapt the base bitrate target dynamically, based on the characteristics of the sequence.
* *`rc_pcrf_base_rc_mode=crf`* may also be used, with base codecs that do support CRF (e.g. x264, x265).

To activate the “CBR base” with a fixed bitrate target for the base (not recommended for best visual quality), the following command should be used: *`rc_pcrf_base_reconfig_mode=0`*.

The FFmpeg command for a 2 Mbps CBR, with default CBR base, is as follows:

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -b:v 2000k -eil_params output.ts`

The FFmpeg command for a 2 Mbps CBR, with CRF base, is modified as follows:

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -b:v 2000k -eil_params "rc_pcrf_base_rc_mode=crf" output.ts`

#### pCRF, capped and uncapped

pCRF, which is the V-Nova LCEVC equivalent of x264’s CRF, aims to ensure that a certain quality factor is maintained throughout the clip. Lower pCRF values mean less compression and higher quality, at the expense of larger file sizes. The pCRF value is a floating-point fractional number with a meaning similar to x264’s CRF (e.g. typical value range 20-36), controlling the overall quality of base + enhancement. To activate pCRF, the following command is used, where X is the chosen pCRF value:\
**rc\_pcrf=X**

**pCRF uncapped**: For uncapped pCRF, the bitrate parameter must be set to 0 or left unspecified, as per the following command line example, in which pCRF 30 is specified.&#x20;

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 [-b:v 0] -eil_params "rc_pcrf=30" output.ts`

**pCRF capped**: For capped pCRF, the bitrate parameter must be set to the desired maximum bitrate (or cap), as per the following command line example, in which pCRF 30 is specified with a maximum of 3,000kbps (3Mbps). Specifying the LCEVC minimum bitrate to 0 will allow the base to use the entire cap if required. The base mode must also be set manually to CRF in this scenario.&#x20;

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 [-b:v 3000] -eil_params "rc_pcrf=30;rc_pcrf_min_bitrate=0;rc_pcrf_base_rc_mode=crf" output.ts`

**QP Min**

The QP (Quantization Parameter) controls the amount of compression for every macroblock in a frame. Large values mean that there will be higher quantization, more compression, and lower quality. Lower values mean the opposite. The full range available for QP is 0-51.

&#x20;By setting the min QP value this will stop the rate control using a value below X. This needs to be done within the `eil_params`.&#x20;

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "rc_pcrf_base_min_qp=14;" output.ts`

**QP Max**

Setting the max QP value will stop the rate control using a value above X. This needs to be done within the `eil_params`.&#x20;

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "qp-max=32" output.ts`

**Important note**: Not all base plugins support this field.


# Lower Resolutions (720p and below)

### General concepts

Even though intermediate resolutions such as 936p are feasible, typical V-Nova LCEVC resolutions below 1080p most frequently range from 720p down to 360p. Resolutions lower than 360p are not recommended, even for extremely low bitrates, e.g. <50 kbps. For these bitrates, it is preferable to employ a lower frame rate; for example, 360p at 7-15 fps is recommended. As illustrated above, for resolutions lower than 540p and relatively high-quality points (e.g. proximity to “convex hull”), 1D mode is generally recommended.

In general, for a given bitrate, V-Nova LCEVC \[x264] allows retaining a higher resolution vs. that of the corresponding native codec \[x264]. For example, for mobile use cases and bitrates lower than 800 kbps, the following bitrate ranges are suggested, based on V-Nova LCEVC x264 -preset slow.

| **V-Nova LCEVC resolution** | **Frame rate** | **Bitrate range** |
| --------------------------- | -------------- | ----------------- |
| 720p                        | 25-30          | 500 – 1000 kbps   |
| 544p                        | 25-30          | 350 – 700 kbps    |
| 480p                        | 25-30          | 150 – 500 kbps    |
| 360p                        | 15-25          | 50 – 250 kbps     |

V-Nova LCEVC settings for lower resolutions are typically in line with the recommendations of the previous sections. The quality of V-Nova LCEVC-enhanced encoding at low bitrates depends mostly on determining the best combination of bitrate, resolution and frame rate for the specific content.


# V-Nova LCEVC-specific Parameters

As with all encoders, additional parameters are available to tune performance optimally for a use case. To help ensure high-quality output, when using LCEVC, the encoder selects, by default, the appropriate parameters for various combinations of bitrate and resolution. Automatic parameter selection can be overridden by the command line.

### LCEVC tuning

**lcevc\_tune**

In line with x264 “tunes,” there are six variants of lcevc\_tune, according to the aim of the encodes. Depending on the chosen tuning, the encoder will combine optimal settings and parameters according to that goal. The settings are as follows:

| **lcevc\_tune setting** | **Description**                                                   |
| ----------------------- | ----------------------------------------------------------------- |
| vq                      | optimizes for visual quality. **Default.**                        |
| vmaf                    | optimizes for VMAF                                                |
| vmaf\_neg               | optimizes for the new VMAF NEG (No Enhancement Gain)              |
| psnr                    | optimizes for PSNR                                                |
| ssim                    | optimizes for SSIM, MS-SSIM                                       |
| animation               | an alternative to 'vq', optimizes for visual quality of animation |

\
As explained in 1.3, please make sure to decode LCEVC streams with the LCEVC-enabled decoder, otherwise the LCEVC data will be ignored and you will decode in backward-compatibility mode. Also, if you are computing objective metrics, please remember to disable dithering at the decoder, as explained in 4.2.5

Examples of command lines:

**CBR, lcevc\_tune vq:**

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -r 30 -g 60 -b:v 1000k -eil_params “preset=medium” lcevc_x264_500k_vq.mp4`

**CBR, lcevc\_tune vmaf:**

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -r 30 -g 60 ‑b:v 1000k -eil_params "lcevc_tune=vmaf;preset=medium" lcevc_x264_500k_vmaf.mp4`

**CBR, lcevc\_tune psnr:**

`ffmpeg.exe -y -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -r 30 -g 60 ‑b:v 1000k -eil_params "lcevc_tune=psnr;preset=medium" lcevc_x264_500k_psnr.mp4`

**Uncapped pCRF, lcevc\_tune vmaf:**

`ffmpeg.exe -y -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -r 30 -g 60 ‑eil_params "rc_pcrf=27;lcevc_tune=vmaf;preset=medium" lcevc_x264_pcrf27_vmaf.mp4`

**Note**: the x264 -preset medium is called out specifically in this command line. However, please be aware that if not specified, this build of FFmpeg will default to medium.&#x20;

### LCEVC preset

**lcevc\_preset**

Since SDK V 3.5 “`lcevc_preset`” is supported. Similar to the `-preset` or `-cpu-used` configuration of other codecs (i.e. x264, x265, VPx, AV1), `lcevc_preset` provides six discrete combinations of encoding parameters to optimise speed and video quality trade-off depending on the use case. The options are from 0 to 5, where 0 is the slowest (i.e., consistently with VPx/AV1) while achieving the maximum quality, and 5 is the fastest with 1 as default.

The “`lcevc_preset`” should be manually set by the user with similar criteria used for the choice of the base encoder `preset`. Here is our recommendation of `lcevc_preset` according to the `preset` of the x264/x265 base encoder. The ‘relative speed index’ provides a ballpark indication of the relative encode time of the various presets vs. the default one; it’s based on LCEVC x264 (medium), an index of ‘110’ means that the given preset takes 10% more encode time vs. the default, while ‘60’ means 40% less encode time. Note: the ratio is approximated and based on a small set of 1080p-encoded sample clips, so results may vary according to testing conditions.

| **lcevc\_preset** | **Example x264/ x265 base preset** | **Example VPx/ AV1 base cpu-used/ preset** | **Relative speed index (based on LCEVC x264)** | **Use case**                                                                                                                                                                                                                                  |
| ----------------- | ---------------------------------- | ------------------------------------------ | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0                 | slow                               | <=5                                        | 110                                            | Maximum quality, to be used when encoding processing is not a major constrain. LCEVC remains much faster than the base codec used alone at full resolution.                                                                                   |
| **DEFAULT**       | **medium**                         | 6-8                                        | **100**                                        | **Optimal speed-quality tradeoff for most use cases where there are no particular speed constraints.**                                                                                                                                        |
| 2                 | fast                               | >8                                         | 90                                             | Further speed gain with negligible quality drop (below JND).                                                                                                                                                                                  |
| 3                 | faster                             | n.a.                                       | 80                                             | Further speed gain, switching off certain perceptual improvements (“priority map”); negligible impact on metrics, but may be visually noticeable on some contents.                                                                            |
| 4                 | very fast, super fast              | n.a.                                       | 65-70                                          | Material speed gain, thanks to switching off LCEVC’s temporal component. Typically not recommended, visual quality can be affected especially on contents with significant static elements (e.g., logos/graphics, eGames, video conferencing) |
| 5                 | ultra fast                         | n.a.                                       | 50-60                                          | Maximum speed gain, but with material impact on visual quality. Not recommended, unless in most extreme scenarios where maximising encoding speed is the main goal.                                                                           |

## Scaling mode

#### **scaling\_mode\_level0**

Specifies the scaling mode for the base encoder picture in the LCEVC hierarchy. In combination with the associated rate control strategies, 2D, 1D and 0D influence the relative allocation of bitrate to the low-, medium- and high-frequency portions of the content. Additional controls, not described in this manual, are available to advanced users.

| **Scaling mode** | **Description**                                                                                                                                                                                                                                                                                                                                                           |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2D               | <p>Two-dimensional 2:1 scaling. E.g. for a 1920x1080 video, base layer is 960x540. <br><strong>Default for resolutions of 720p and above.</strong></p>                                                                                                                                                                                                                    |
| 1D               | <p>Horizontal-only 2:1 scaling. E.g. for a 1920x1080 video, base layer is 960x1080. This mode is recommendable at high bits per pixel (e.g. full HD above 5 Mbps) or low resolutions (e.g. 540p or below), especially for content with high amounts of relatively low-contrast high-frequency detail.</p><p><strong>Default for resolutions lower than 720p.</strong></p> |
| 0D               | No scaling. LCEVC enhances a base at the same resolution. This mode is primarily used to modify video features with LCEVC such as providing an HDR output with an SDR base. 0D mode must also be used when encoding in [Native mode](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#encoding-mode) (without LCEVC).                                   |

2D is generally recommended for HD and UHD content and is the default scaling mode setting. 1D instead is the recommended mode, and default setting, for lower resolutions (540p and below).

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "scaling_mode_level0=2D" output.ts`

In certain cases (e.g. at high bits per pixel for HD/UHD, or at medium bits per pixel for lower resolutions), 1D may provide a preferable trade-off between robustness to banding/blocking vs. loss of resolution impairments, especially when the content is viewed on a large display or from viewing distances lower than 2H.

Especially when comparison of different encoding parameters is possible (e.g. Convex Hull approach for VOD dynamic optimization, CAE, etc.), we recommend to also include 1D as option for high-bitrate and low-resolution profiles. At relatively low bits per pixel, 2D will provide better protection from big impairments during complex scenes, whilst possibly generating some loss of resolution in low-contrast areas during low-to-medium complexity scenes. Vice versa for 1D scaling mode.

## Encoding Mode

#### **encoding\_mode**

Specifies if LCEVC enhancement or native coding is applied.

| **Enhancement setting** | **Description**                                                                                                                                                                                  |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| enhanced                | The enhancement coding process is applied. **Default.**                                                                                                                                          |
| native                  | <p>Pass-through. Only the leveraged codec (e.g. x264) is used in full resolution, with no LCEVC enhancement. To be used in combination with:</p><p><strong>scaling\_mode\_level0=0D</strong></p> |

Example:

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "scaling_mode_level0=0D;encoding_mode=native" output.ts`

Note: when using encoding\_mode=native, the LCEVC enhancement is effectively switched off. In this mode, it is recommended NOT to specify any LCEVC specific parameters in the eil\_params string to avoid unexpected behaviour or erroring.

## Dithering

Specifies custom dither. Dither is an intentionally applied form of subtle noise / camera grain used in constrained bandwidth conditions, to minimise visual impairments, such as colour banding or blocking artefacts due to a constrained base layer.

For some scenes and types of content, dithering can provide a significant uplift in perceived quality, although objective metrics will always be worse when dithering is active. For this reason, dithering is turned off by default for all lcevc\_tune settings except lcevc\_tune vq.

Below is an example (with gamma increased to 3.0 in order to highlight the effect) of how dithering reduces aliasing on edges and reduces banding/blocking impairments. The effect is even more pleasant in motion, since banding and blocking may follow motion patterns distinct from the object that they overlay.

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/nODFPXdk8LpyPeuLaZmg/3.png)![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/ms5ojltETQryyNUH9Z6k/4.png)

Figure 4.1 — Same LCEVC encode (dark scene, gamma adjusted to 3.0), decoded with adaptive dithering off (i.e., ignoring the dithering signalling) vs. adaptive dithering on.

#### **dc\_dithering\_type**

Specifies whether to apply a uniform dithering algorithm.

| **Dither setting** | **Description**                                                           |
| ------------------ | ------------------------------------------------------------------------- |
| None               | No dithering is applied. **Default for lcevc\_tune psnr, vmaf and ssim**. |
| Uniform            | Uniform random dithering applied. **Default for lcevc\_tune vq**.         |

#### **dc\_dithering\_strength**

Specifies the maximum dithering strength. Dithering preferences are often subjective:

* The default value is **4**.
* A value of 7-8 displays a more visible dither.
* A value of 2-3 should be used for substantially imperceptible dither.

Dithering is applied dynamically and content-adaptively by the encoder, depending on the quality of the base layer (base qp), on the level of lighting of a scene as well as on other factors. Irrespective of the specified strength, it automatically disappears in static/low-motion, low-detail scenes, and its intensity is automatically modulated on a frame-by-frame basis, according to the base QP, starting above a certain threshold (dc\_dithering\_qp\_start), and maxing out above a second threshold (dc\_dithering\_qp\_saturate). When dithering is activated, also at low base QP some dithering may still be applied by the encoder in case of dark scenes with relatively noisy source content.

#### **dc\_dithering\_qp\_start**

This parameter specifies the base QP value at which to start applying dither. **Range: 0-51**. **Default:** **24**.

#### **dc\_dithering\_qp\_saturate**

This parameter specifies the base QP value at which to saturate dither. **Range: 0-51**. **Default:** **36**.

Regardless of the base QP value, other low-level parameters make dithering adapt dithering strength settings based on frame luminosity (according to contrast sensitivity function) as well as presence of no-contrast plain graphics which would not benefit from dithering.

#### Dithering at the decoder

V-Nova recommends that dithering be used for optimal subjective visual quality. However, when calculating objective metrics for content that was encoded with dithering enabled, dithering must be disabled via the following command:

#### **disable\_dithering**

Enables/disables the dithering algorithm (see section 8 for use with an FFmpeg decoding command line).

| **Disable dithering setting** | **Description**                                                                                                                 |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| 0                             | Dithering setting is unaffected, i.e., it is performed adaptively as indicated within the LCEVC elementary stream. **Default**. |
| 1                             | Dithering is disabled.                                                                                                          |

As an example:

`ffmpeg.exe -vcodec lcevc_h264 -disable_dithering 1 -i stream.mp4 -vcodec rawvideo output.yuv`

## M Adaptive Downsampling&#x20;

The M Adaptive Downsampling settings influence a combination of LCEVC advanced encoder settings that affect the interlocked image processing effects of downsampler filter, upsampler filter, predicted residuals and full-resolution residual data. The LCEVC format is extremely flexible, allowing the encoder to leverage both non-linear (as well as content-adaptive) downsampling methods and signal custom (content- or context- adaptive) upsampling kernels. The upsampled reconstruction, before adding full-resolution details, is further amended by the LCEVC decoder with a non-linear processing step called “Predicted Residuals”. The combination of these cascade of non-linear adaptive filters generates a sort of simplified super-resolution upsampling, which is further corrected by adding details (residual data) that could not be otherwise reconstructed, so as to approximate the source as closely as possible, up to mathematically lossless.&#x20;

The overall compression efficiency of LCEVC-enhanced multi-layer coding vs. the enhanced single-layer codec used alone at full resolution comes from sensibly separating high-frequency energy (“details”) from medium-to-low frequency energy (“core signal”), and efficiently compressing both components of the signal with:

a) low-complexity tools specifically designed to efficiently compress sparse high-frequency details via light-weight parallel processing

and&#x20;

b) a traditional single-layer codec operating more efficiently on the core signal by compressing it at a lower resolution.

Modifying some key elements of this non-linear combination of resampling and signal decomposition tools generates profound impact to both visual quality and metrics. In the current implementation we established some combinations that work reasonably well, and we embedded them in the various lcevc\_tunes. But the calibration effort isn’t infallible: we observed material divergence in how different objective metrics (as well as subjective preferences) react to changes in these low-level settings. In short, there is still much room for improvement and fine-tuning. Future releases will further improve the way in which the encoder automatically calibrates these tools based on user preference and on the specificity of the content being encoded.

The M adaptive downsampling (`m_ad_mode`) settings deviate from basic linear kernels and provide some degree of control for one of the elements of this “chain reaction” of interlocked non-linear image processing tools.

#### **m\_ad\_mode**

Specifies the M adaptive downsampling mode (String).

| Mode     | **Description**                                                                                                                                                       |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| disabled | <p>M adaptive downsampling disabled. </p><p><strong>Default for lcevc\_tune=psnr, lcevc\_tune=ssim</strong> and <strong>lcevc\_tune=vmaf\_neg</strong></p>            |
| replace  | <p>M adaptive downsampling is applied equally to both residual surfaces. </p><p><strong>Default for lcevc\_tune=vq</strong> and <strong>lcevc\_tune=vmaf</strong></p> |
| separate | <p>M adaptive downsampling is applied separately to residual surfaces.</p><p>Default for <strong>lcevc\_tune=animation</strong></p>                                   |

Notice: MSE-based metrics such as PSNR and SSIM strongly “dislike” the use of M adaptive downsampling, so if you are in any way looking at any MSE-based metrics, either set `m_ad_mode=disabled` or use the corresponding `lcevc_tune` (which, among other things, will set m\_ad\_mode to "disabled). On the other side, both formal subjective MOS scores and VMAF tend to agree that some degree of M adaptive downsampling improves visual quality.

#### **m\_hf\_strength**

m\_hf\_strength, which accepts fractional values between 0 and 0.5, allows to increase or decrease the energy of high frequencies, with 0 being a preference for softer details. Default values, which are modified adaptively by the encoder if you do not specify anything, are comprised between 0 and 0.35.

#### **m\_lf\_strength**

m\_lf\_strength, which accepts fractional values between 0 and 1.0, allows you to modify the way in which full resolution details are separated from the mid-to-low frequencies that are passed as low resolution to the base codec. Default values, which are modified adaptively by the encoder if you do not specify anything, are comprised between 0 and 0.5.

## IPP mode (no b-frames)

For certain low-latency applications, such as video conferencing, the V-Nova LCEVC rate control includes a specific IPP mode, i.e., in which b-frames are not used. This mode must be turned on for optimal performance, in combination with setting b-frames to zero (bframes=0 in the case of x264); otherwise the V-Nova LCEVC rate control will make incorrect assumption about the GOP structure, and consequently, will make suboptimal rate allocations.

**rc\_pcrf\_ipp\_mode**

Specifies whether to apply IPP mode.

| **IPP mode setting** | **Description**                                                      |
| -------------------- | -------------------------------------------------------------------- |
| 0                    | No IPP mode applied, i.e., assumption of IBP structure. **Default**. |
| 1                    | IPP mode applied.                                                    |

Example:

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "bframes=0;rc_pcrf_ipp_mode=1" output.ts`

## Optimising Encoding CPU utilization&#x20;

There are two main tools we can use for CPU optimisation: 1; NUMA awareness, 2; api mode.\
\
**NUMA**\
For best performance it is crucial to make sure that a process does not cross NUMA nodes or physical CPU sockets as the LCEVC SDK is currently not NUMA aware. This is most relevant when using physical server hardware or large compute cloud environments. \
\
Linux: run 'lscpu'\
Windows: Open Task Manager > Performance > right click > change graphs to NUMA node, if grey your system only has 1 node.&#x20;

```
NUMA node0 CPU(s):   0-17,36-53
NUMA node1 CPU(s):   18-35,54-71          
```

Here is an example of the NUMA information from a large Linux server. \
\
If we are to run encodes in parallel, job A would be best on NUMA 0,  job B on NUMA 1 and so on. This is to stop frame data having to transfer between nodes which is a bottleneck for encoding. \
\
This can be achieved by using a tool such as 'taskset' on Linux or 'AFFINITY' on Windows to restrict the cores that the function is allowed to run on. \
\
**API Mode**\
Another method is to add the following to the eil parameters within your LCEVC encode. This is system dependent and should be used on larger systems, but can also have a negative impact on encoding performance on smaller machines (below 16 threads) where it should not be used.

```
eil_params= "api_mode=asynchronous" 
```

This parameter decouples the encoding pipeline into different queues rather than processing all in one queue across the threads.&#x20;

## Querying Configuration Properties

All publicly available encoding configuration properties for the base encoder and LCEVC enhancement can be queried through the API using `EIL_QueryPropertyGroups`. This will return an array of all property groups which each contain an array of all properties in that group. Alternatively, if the name of the parameter is known then it can be directly queried through `EIL_QueryProperty`.&#x20;

Each property contains a name, description, the expected type of that property and its current value. Below is an example that queries the API for all properties and prints them.&#x20;

```c
void PrintProperties(EILContext context)
{
	EILPropertyGroups groups;
	EILReturnCode     rc = EIL_QueryPropertyGroups(context, &groups);

	if (rc != EIL_RC_Success)
	{
		fprintf(stderr, "Unable to query property groups: %s\n", EIL_GetErrorString(rc));
		return;
	}

	for (uint32_t g = 0; g < groups.group_count; ++g)
	{
		EILPropertyGroup* group = &groups.group[g];

		printf("%s:\n", group->name);

		for (uint32_t p = 0; p < group->property_count; ++p)
		{
			EILProperty* property = &group->properties[p];

			printf("\t%s (", property->name);

			switch (property->type)
			{
				case EIL_PT_Int16: printf("int16"); break;
				case EIL_PT_Int32: printf("int32"); break;
				case EIL_PT_Int64: printf("int64"); break;
				case EIL_PT_Uint16: printf("uint16"); break;
				case EIL_PT_Uint32: printf("uint32"); break;
				case EIL_PT_Uint64: printf("uint64"); break;
				case EIL_PT_Float: printf("float"); break;
				case EIL_PT_Double: printf("double"); break;
				case EIL_PT_String: printf("string"); break;
				case EIL_PT_Boolean: printf("boolean"); break;
				case EIL_PT_Blob: printf("blob"); break;
			}

			printf("): %s\n", property->description);
		}
	}
}
```


# Metadata

## Overview

Metadata works much in the same way as properties, except that they are queried only from the system, and they cannot be set externally. They can be treated as read-only properties.

Metadata can only be queried by name using the `EIL_QueryMetadata` API.

Metadata additionally implements an extension to properties allowing access to a contiguous block of memory (`EIL_PT_Blob`). This system is incorporated with the Base encoder plugin, such that a Base encoder can output some data to the integrator of the IL, with the IL being ignorant to its usage.

This system is used to allow the EIL to report to the user some internal values, or data that are either not interpreted as properties, or that cannot be expressed as properties.

For example, the ffmpeg sample integration utilises this feature to obtain the h.264 global headers from the libx264 Base encoder plugin to be used for HLS.

## Metadata Values

### Overview

This section describes the metadata values available from the EIL. It is broken down into sections for each discrete component of the EIL. The following tables indicate the name of the metadata, its associated property type, and a description of what it is and how it is used.

### Integration Layer

| **Metadata** | **Type**        | **Description**                                                                                                                             |
| ------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| base\_type   | EIL\_PT\_String | Describes the underlying technology of the base encoder plugin utilised. For example, the libx264 plugin will output h264 in this metadata. |
| base\_width  | EIL\_PT\_Uint32 | Width of the base encoded video.                                                                                                            |
| base\_height | EIL\_PT\_Uint32 | Height of the base encoded video.                                                                                                           |

### x264 Integration Layer Plugin

<table data-header-hidden><thead><tr><th width="196.33333333333331">Metadata</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><strong>Metadata</strong></td><td><strong>Type</strong></td><td><strong>Description</strong></td></tr><tr><td>h264_header_data</td><td>EIL_PT_Blob</td><td>This is available when the ‘global-header’ x264 property is set to 1. This is a contiguous block of memory containing the H.264 headers (PPS, SPS, etc.), and can be used when the output container is segmented, so that the headers can be injected at the start of each segment.</td></tr></tbody></table>


# API (version 3.9)

## EIL\_GetErrorString

```c
VN_EIL_PublicAPI const char* EIL_GetErrorString(EILReturnCode error);
```

Utility method for getting a string representation of an EIL return code in English.

## EIL\_OpenSettingsDefault and EIL\_Open

### EIL\_OpenSettingsDefault

```c
VN_EIL_PublicAPI EILReturnCode EIL_OpenSettingsDefault(EILOpenSettings* settings);
```

Initialises the members of an `EILOpenSettings` object to sensible default state. Some fields of the settings must still be populated by the integration before the settings can be used by the EIL.

### EIL\_Open

```c
#define VN_EIL_OpenFnName VN_EIL_PP_Concat(EIL_Open_, VN_EIL_API_Version)

VN_EIL_PublicAPI EILReturnCode VN_EIL_OpenFnName(EILOpenSettings* settings, EILContext* output);

#define EIL_Open VN_EIL_OpenFnName
```

This is the entry point for creating an instance of the EIL. The open function will have the version of the API appended at the end of the function name to ensure that a newer or older version of the EIL that contains breaking changes cannot be mistakenly used. As such, the `EIL_Open` define is provided to aid in calling the correct open function.

If the EIL fails to open, then it is typically due to not being able to find and open the requested base encoder. A detailed message as to why the encoder failed to open should be sent to the log callback, if it is provided.

### EILContext

```c
typedef struct EILContextImpl* EILContext;
```

The `EILContext` is an opaque handle to the EIL's encoder instance. An `EILContext` is created when the EIL is opened using `EIL_Open`, and the same instance should be passed to all other functions. It is destroyed when `EIL_Close` is called.

### EILOpenSettings

#### ***EILOpenSettings Overview***

```c
typedef struct EILOpenSettings
{
    const char* base_encoder;
    void*       base_encoder_userdata;
    void        (*log_callback)(void* userdata, int32_t level, const char* msg);
    void*       log_userdata;
    const char* log_filepath;
    void*       reserved;
    void*       reserved2;
} EILOpenSettings;
```

`EILOpenSettings` contains the information required to open the EIL. It can be default initialised using `EIL_OpenSettingsDefault`.

#### ***base\_encoder***

```c
    const char* base_encoder;
```

\[ Required ]

The name of the base encoder that the EIL uses for base encoding should not contain the preceding `lcevc_eilp_` (`liblcevc_eilp_` on Linux) or the platform's dynamic library extension. For example, if a plugin is called `liblcevc_eilp_x264.so` or `lcevc_eilp_x264.dll`, then the `base_encoder` string should be `x264`.

There are two special strings for `base_encoder`: `none` and `inject`:

* `none` is a pass-through plugin that sets the output recon to the input.
* `inject` allows for `reserved2` to reference an `EILPAPI` object that contains function pointers for all of a plugin's methods.

#### ***base\_encoder\_userdata***

```c
    void*       base_encoder_userdata;
```

\[ Dependent on the plugin | Default: `NULL` ]

The `base_encoder_userdata` is a handle that allows some arbitrary data to be passed through to a base plugin.

#### ***log\_callback***

```c
    void        (*log_callback)(void* userdata, int32_t level, const char* msg);
    void*       log_userdata;
```

\[ Optional | Default: `NULL` ]

`log_callback` can be set to a method that receives all log messages.

`userdata` is the value set in `log_userdata`, and `level` is one of the levels in the enumeration `EILLogLevel`. If left unset, then all logging is disabled.

All log messages are in the format `level | group | message\n`, and are limited to 4096 bytes.

#### ***log\_filepath***

```c
    const char* log_filepath;
```

\[ Optional | Default: `NULL` ]

`log_filepath` is an optional file path, that when specified , contains all log messages.

#### ***reserved, reserved2***

```c
    void*       reserved;
    void*       reserved2;
```

Reserved fields that should not be modified, unless specified elsewhere.

## EIL\_InitSettingsDefault and EIL\_Initialise

### EIL\_InitSettingsDefault

```c
VN_EIL_PublicAPI EILReturnCode EIL_InitSettingsDefault(EILInitSettings* settings);
```

Initialises the members of an `EILInitSettings` object to sensible defaults. Some fields of the structure must still be populated by the integration before the settings can be used by the EIL.

### EIL\_Initialise

```c
VN_EIL_PublicAPI EILReturnCode EIL_Initialise(EILContext context, EILInitSettings* settings);
```

Initialises the EIL so that it is ready to receive pictures for encoding. The context must have previously been created using `EIL_Open`.

## EILInitSettings

### EILInitSettingsOverview

```c
typedef struct EILInitSettings
{
    EILColourFormat color_format;
    uint32_t        width;
    uint32_t        height;
    uint32_t        fps_num;
    uint32_t        fps_denom;
    uint32_t        bitrate;
    int32_t         gop_length;
    const char*     properties_json;
    uint8_t         external_input;
} EILInitSettings;
```

`EILInitSettings` contains the information required to initialise the EIL. It can be default initialised using `EIL_InitSettingsDefault`.

#### color\_format

```c
    EILColourFormat color_format;
```

\[ Optional | Default: `EIL_YUV_420P` ]

The colour format of the *input* image. If it is not one of the YUV formats, then there will be an extra conversion step that converts the input into the YUV format. The matrix used for the colour conversion can be controlled using the `picture_colorspace` config option.

#### width, height

```c
    uint32_t        width;
    uint32_t        height;
```

\[ Required ]

The resolution of the input image. Once the EIL is initialised. then these are modified by the EIL to become the resolution of the base video. These modified values should be used by any operation or process that consumes the bitstream. Consult the LCEVC standard for supported resolutions.

#### fps\_num

```c
    uint32_t        fps_num;
    uint32_t        fps_denom;
```

\[ Required ]

The framerate's numerator and denominator, e.g., `25/1` for 25fps or `30000/1001` for 29.97fps

#### bitrate

```c
    uint32_t        bitrate;
```

\[ Required ]

The target bitrate, in *kilo*bits/s, at which to encode.

#### gop\_length

```c
    int32_t         gop_length;
```

\[ Optional | Default: `EIL_GOPLengthAuto` ]

The GOP length in frames. There are three values to which `gop_length` can be set, subject to the base encoder's abilities:

* `EIL_GOPLengthAuto`: The EIL determines which GOP length to use.
* `EIL_GOPLengthInfinite`: Only the first frame is an I-frame.
* `EIL_GOPLengthIntraOnly`: Every frame is an I-frame.

#### properties\_json

```c
    const char*     properties_json;
```

\[ Required ]

A string containing json properties for the EIL's user-specified options.

#### external\_input

```c
    uint8_t         external_input;
```

\[ Optional | Default: 0 ]

Whether or not the input image's planes should be allocated by the integration or the EIL.

* When set to `0`, the EIL allocates and manages the lifetimes of an image's planes.
* When set to `1`, the planes are expected to be allocated and managed external to the EIL.

## EIL\_GetPicture

### EIL\_GetPicture Overview

```c
VN_EIL_PublicAPI EILReturnCode EIL_GetPicture(EILContext context, EILFrameType frameType, EILPicture** picture);
```

When the EIL is configured to operate in internal input mode (see `external_input` under `EIL_InitSettings`), this method is used to obtain a picture object with preallocated planes. The planes can then be populated with frame data and passed to `EIL_Encode`. A picture obtained using this method is owned by the integration until that picture is passed back through `EIL_Encode`.

The EIL internally creates a pool of pictures from which this method extracts. The most common reason for the failure of this method, is that the pool of pictures is empty. This can occur if this function is called too frequently without returning the pictures through `EIL_Encode`. The size of the pool depends on the base encoder's frame delay plus the EIL's frame delay, where frame delay is the number of frames that must be provided to the encoder before any output is produced.

## EILPicture

### ***EILPicture Overview***

```c
typedef struct EILPicture
{
    EILMemoryType   memory_type;
    uint32_t        num_planes;
    void*           plane[EIL_MaxPlanes];
    uint32_t        stride[EIL_MaxPlanes];
    EILBaseType     base_type;
    EILFrameType    frame_type;
    EILFieldType    field_type;
    int64_t         pts;
    void*           user_data;
    const void*     reserved;
} EILPicture;
```

An `EILPicture` is in the input to the encoder, containing all information needed to encode a single frame of video. If the EIL is configured to operate in external input mode, then it is up to the integration to set the fields of this struct, in which case it can be default initialised using `EIL_InitPictureDefault`. Otherwise, the EIL will populate the fields of the struct, and the integration should populate the preallocated planes.

***memory\_type***

```c
    EILMemoryType   memory_type;
```

\[ Optional if externally allocated | Default `EIL_MT_Host` ]

`memory_type` specifies the type of memory stored in the picture's planes. This can currently only be either:

* `EIL_MT_Host` is memory that is accessible from the device's CPU, i.e., allocated using `malloc` or some device memory that has been mapped. If the memory is externally allocated, then the memory must be aligned for 8- or 16-bit access, depending upon the bitdepth.
* `EIL_MT_AHardwareBuffer` indicates that the planes are Android hardware buffers

  (`AHardwareBuffer`). The encoder must be configured to use the GPU for this to be accepted, when used in conjunction with external input.

Other memory types:

* `EIL_MT_TextureGL` is currently unsupported as input to the EIL. The EIL *may* pass this type of

  memory to the base encoder, depending upon the configuration.
* `EIL_MT_Unknown` is used only to indicate that the memory type is not yet defined. `memory_type` should not be set to `EIL_MT_Unknown` by the point of ingest.

***num\_planes, plane\[EIL\_MaxPlanes]***

```c
    uint32_t        num_planes;
    void*           plane[EIL_MaxPlanes];
```

\[ Required if externally allocated ]

Pointers to the appropriate structures holding the plane data, as defined by the `memory_type`. These planes can be interleaved in the cases of the RGB formats. As such, `num_planes` indicates the number of plane pointers rather than the number of channels. In the case of contiguous memory, each pointer should be set to the starting address of each plane, and the number of planes should be set as if the memory is not contiguous.

***stride\[EIL\_MaxPlanes]***

```c
    uint32_t        stride[EIL_MaxPlanes];
```

\[ Required if externally allocated ]

The stride, in bytes, for each plane defined in `plane`.

***base\_type***

```c
    EILBaseType     base_type;
```

\[ Optional | Default: `EIL_BT_Uknown` ]

If set to a value other than the default, then when this picture is encoded, the EIL will attempt to force the requested type of picture from the base encoder. Whether or not the requested type of frame is produced, is subject to a base encoder's capabilities.

***frame\_type, field\_type***

```c
    EILFrameType    frame_type;
    EILFieldType    field_type;
```

\[ Optional if externally allocated | Default: `EIL_FrameType_Progressive` ]

Whether this frame is progressive, a single field, or interlaced, and if it is a single field, then whether the frame is the top or bottom field.

***pts***

```c
    int64_t         pts;
```

\[ Optional | Defaut: `-1` ]

The presentation timestamp of the picture. If left as the default, then it is determined by the EIL.

***user\_data***

```c
    void*           user_data;
```

\[ Optional | Default: `NULL` ]

A handle to some arbitrary data that is set on the `EILOutput` produced by the picture.

***reserved***

```c
    const void*     reserved;
```

Internally used field. Do not modify.

## EIL\_Encode

```c
VN_EIL_PublicAPI EILReturnCode EIL_Encode(EILContext context, EILPicture* picture);
```

Passes a picture that is ready for encode to the EIL. A picture can be obtained from the EI, either:

* Using `EIL_GetPicture`, if external input is disabled; or,
* Created and populated by the integration, if external input is enabled.

## EIL\_GetOutput and EIL\_ReleaseOutput

### Get\_Output

```c
VN_EIL_PublicAPI EILReturnCode EIL_GetOutput(EILContext context, EILOutput** output);
```

Gets the next output available from the EIL. The integration has ownership of the `EILOutput` until it has been passed back through `EIL_ReleaseOutput`. This can return one of two return codes upon success:

* `EIL_RC_Success`: Successfully got the next frame of output.
* `EIL_RC_Finished`: No errors occurred but there there is currently no output available.

### Release\_Output

```c
VN_EIL_PublicAPI EILReturnCode EIL_ReleaseOutput(EILContext context, EILOutput* output);
```

Returns an output obtained using `EIL_GetOutput` back to the encoder.

***EILOutput***

* **EILOutput Overview**

```c
typedef struct EILOutput
{
    EILFrameType    frame_type;
    EILFieldType    field_type;
    EILBaseType     base_type;
    int8_t          keyframe;
    int32_t         qp;
    int64_t         pts;
    int64_t         dts;
    int8_t          config;
    const uint8_t*  data;
    uint32_t        data_length;
    const uint8_t*  lcevc;
    uint32_t        lcevc_length;
    void*           user_data;
    const void*     reserved;
} EILOutput;
```

The `EILOutput` struct contains all information for an encoded frame of video. These should never have to be created manually, and should be treated as read-only.

* **frame\_type, field\_type**

```c
    EILFrameType    frame_type;
    EILFieldType    field_type;
```

Whether the frame that produced this output is progressive, interlaced or a field. `field_type` is applicable, only if the type is `EIL_FT_Field`.

* **base\_type, keyframe**

```c
    EILBaseType     base_type;
    int8_t          keyframe;
```

The type of frame that the base encoder produced, and whether or not it is flagged as a keyframe. Refer to the appropriate base encoder's standard for their meanings.

* **qp**

```c
    int32_t         qp;
```

The Quantisation Parameter (QP) of the frame, as reported by the base encoder.

* **pts**

```c
    int64_t         pts;
```

The presentation timestamp of the encoded frame. This is the input frame's pts, if it was specified; otherwise, it is the pts that the base encoder produced.

* **dts**

```c
    int64_t         dts;
```

The decode timestamp of the encoded frame, as produced by the base encoder.

* **config**

```c
    int8_t          config;
```

Whether the data in this output is the encoder's config. For example, if the base encoder is an H264 encoder and `config` is `1`, then the output contains only the SPS and PPS NAL units. The config output is always presented before any frame data.

* **data, data\_length, lcevc, lcevc\_length**

```c
    const uint8_t*  data;
    uint32_t        data_length;
    const uint8_t*  lcevc;
    uint32_t        lcevc_length;
```

The encoded bitstream for the frame and its length. If the `separate_output` config option is set, then the LCEVC data is on the `lcevc` field; otherwise, it is contained in the `data` field, along with the base bitstream.

* **user\_data**

```c
    void*           user_data;
```

The user data that was set on the picture that produced this frame.

* **reserved**

```c
    const void*     reserved;
```

Internally used field. Do not modify.

## EIL\_QueryPropertyGroups,  EIL\_QueryProperty

### EIL\_QueryPropertyGroups, EIL\_QueryProperty Overview

```c
VN_EIL_PublicAPI EILReturnCode EIL_QueryPropertyGroups(EILContext context, EILPropertyGroups* output);
VN_EIL_PublicAPI EILReturnCode EIL_QueryProperty(EILContext context, const char* name, EILProperty* output);
```

This group of methods is used to query the options available through the EIL. These are usually controlled by the user.

* `EIL_QueryPropertyGroups` returns a collection of all properties in their logical groupings.
* `EIL_QueryProperty` obtains an individual property by its name.

The properties and their values are passed to the EIL in JSON format on the `EILInitSettings` structure through the `properties_json` field.

### EILPropertyGroups

```c
typedef struct EILPropertyGroups
{
    EILPropertyGroup*   group;
    uint32_t            group_count;
} EILPropertyGroups;
```

This structure contains an array of all property groups.

```c
typedef struct EILPropertyGroup
{
    const char*     name;
    const char*     description;
    EILProperty*    properties;
    uint32_t        property_count;
} EILPropertyGroup;
```

The property group contains a logical grouping of all properties available through the EIL and base encoders.

## EILProperty

### ***EILProperty Overview***

```c
typedef struct EILProperty
{
    const char*     name;
    const char*     description;
    EILPropertyType type;

    union
    {
        int32_t     i32;
        int64_t     i64;
        uint32_t    u32;
        uint64_t    u64;
        float       f;
        double      d;
        const char* str;
    } value;

    const uint8_t*  blob;
    uint32_t        blob_size;
} EILProperty;
```

This structure is used to describe config properties or metadata of the EIL and the base encoder.

***name, description***

```c
    const char*     name;
    const char*     description;
```

A short name and description of the property that was queried.

***type***

```c
    EILPropertyType type;
```

The type of value contained in this property. This value should be used to determine which field of the `value` union is set:

* `EIL_PT_Int32`: `value.i32`
* `EIL_PT_Int64`: `value.i64`
* `EIL_PT_Uint32`: `value.u32`
* `EIL_PT_Uint64`: `value.u64`
* `EIL_PT_Float`: `value.f`
* `EIL_PT_Double`: `value.d`
* `EIL_PT_String`: `value.str`
* `EIL_PT_Boolean`: `value.i32`

***blob, blob\_size***

```c
    const uint8_t*  blob;
    uint32_t        blob_size;
```

`blob` and `blob_size` are used when the property type is `EIL_PT_Blob`. This is only ever the case when querying metadata.


# NDK


# LCEVC Encoder IP Core

V-Nova's LCEVC Encoder IP core is a versatile solution that supports video resolutions up to 8K at frame rates up to 60 fps. It includes V-Nova's unique pre-processing and rate control enhancements  to deliver exceptional video compression performance. It has been compiled for and verified on Xilinx Alveo U50 and U250 boards, guaranteeing reliability and performance.

By leveraging the LCEVC Encoder IP core, you can easily incorporate LCEVC encode capability into your next generation video SoC and take advantage of its enhanced video coding capabilities.

## **Design Features of the V-Nova LCEVC Encoder IP**

The V-Nova LCEVC Encoder IP core has been designed to address the video encoding needs of different applications and SOC architectures:

**MPEG-5-part2 LCEVC Specification Compliance**

Foremost among these features is the Encoder IP's compliance with the MPEG-5-part2 LCEVC specification (ISO/IEC 23094-2). &#x20;

**Adaptable Integration Architecture**

The Encoder IP has been designed with adaptability in mind. This includes the parametrization of vital metrics such as maximum resolution, performance throughput, latency, logic area, and DDR bandwidth availability. This flexible approach enables integrators to tailor the Encoder IP's configuration to align precisely with their project's unique requirements.

**High Performance**

The Encoder IP design can support a maximum throughput of 8K resolution at 120 frames per second, at 4:2:0 bit depths of 8, 10 or 14.

*Further details available on request.*


# Getting started

There are a broad range of options for playing back LCEVC-enhanced content across the majority of popular devices and platforms. This is facilitated by the low-complexity nature of the standard which enables simple software updates to apps and web pages which use generic CPU or GPU processing to upgrade to LCEVC-enhanced streaming. This is typically achieved without any impact on overall decode processing load or battery consumption compared to natively encoded content.

This page outlines two fundamental paths for testing playback of your LCEVC-enhanced content:

* [Playing back content locally](#playing-back-local-content) (e.g. MP4 files you have created with FFmpeg)
* [Playing back remote packaged content](#playing-back-remote-packaged-content) (e.g. HLS or MPEG-DASH manifests)

## Playing back local content

If you are testing LCEVC-enhanced encoding locally there are several options outlined in this section for playing back either MP4 or MPEG-TS. These options are FFmpeg, ExoPlayer or the V-Nova application on Windows.

### 1.  FFmpeg

#### **Decode LCEVC in ffplay for playback.**

Using the same process as decoding in FFmpeg described in step 4, if decoding for playback using the V-Nova SDK the video codec input type must be specified before the input (see below).

&#x20;We suggest using the test file created in section 3 to confirm the workflow is functioning correctly.

&#x20;`.\ffplay.exe -vcodec lcevc_h264 -i ${INPUT_PATH}`

This will then open a window for preview of the video. If the playback does not include the test enhancement layer with bouncing squares, then LCEVC layer has not been decoded.

Once you have confirmed the process using the test with the bouncing squares, you can then run the encode / decode process with your real video stream to observe the benefits of LCEVC in terms of reduced file size and improved quality etc.

#### **Decode LCEVC in FFmpeg for file conversion or analytics**

To decode a file using the V-Nova SDK, the video codec input type must be added to the command line, before the input file is specified (see below).

We suggest using the file created in section 3 to confirm the workflow is functioning correctly.

\
`.\ffmpeg -vcodec lcevc_h264 -disable_dithering 1`` ``-i ${INPUT_PATH} -c:v rawvideo output_%width_%height.yuv`\
\
If playback of the output YUV file does not include the bouncing squares, then LCEVC layer has not been decoded correctly. Tools such as Vooya or YUV Toolkit can be used to play back the YUV file with additional view and analysis options. FFplay can also be used for simple decoding of YUV files. \
\
Similarly, to run VMAF using ffmpeg, you must specify the encoded file first before stating the reference (source) file.

&#x20;`.\ffmpeg.exe -vcodec lcevc_h264 -disable_dithering 1 -i input_LCEVC_encoded.ts` \
`-s 1920x1080 -pix_fmt yuv420p -i input_source_1920x1080_50.yuv -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl:log_path=VMAF_lcevc_score.txt" -f null -`

### 2. AndroidX (ExoPlayer)

If you have been provided with the LCEVC-enabled ExoPlayer reference integration as part of your evaluation, you can use the pre-built LCEVC ExoPlayer APK to play back both local content or from a URL.

<img src="https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/HMUIeecJFDlEBNynzcNp/Screenshot_20220613-185108_LCEVC-ExoPlayer.jpg" alt="" data-size="original">

![Figure 4 Playback controls of the demo app](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/LaVO2krk3bHgEePByvb8/3.jpeg)

The app can play:

1. Local files, by selecting the GO button with an empty URL field
2. A URL entered in the URL field (e.g. HLS or MPEG-DASH manifest link)

[Go here](/technologies/mpeg-5.lcevc/integrations/integrations/androidx-exoplayer) for more information on ExoPlayer.

### 3. V-Nova App on Windows

V-Nova's demo application on Windows also includes the functionality to play back local files and can be d[ownloaded here from the Microsoft Store](https://www.microsoft.com/en-qa/p/v-nova/9n9c2z21xhl9?cid=msft_web_chart\&activetab=pivot:overviewtab).

To play back local content simply select "Launch Player" from the app's login screen:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/wkDxOQfbi6wjZ91byjNb/image.png)

Once the player has launched click on the browse icon on the app's toolbar to select a local file:

![V-Nova Windows app toolbar](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/6p5j0dDU5lVKJYOXmyDi/image.png)

## Playback HTTP adaptive streaming Content

If you have encoded content and packaged it as HLS or MPEG-DASH for delivery via a CDN you have a variety of options to play back those streams. Please note that if you have encoded content on the V-Nova Platform then you can play back all of your content simply by signing into any of the V-Nova apps - see [Playing back content encoded on the V-Nova platform](#playing-back-content-encoded-on-the-v-nova-platform).

### 1. V-Nova Web Player

Head to [player.v-nova.com](https://player.v-nova.com) and after signing in you can select "Launch Player" from the menu in the top-left of the screen. This will bring up a player that allows you to specify any stream manifest link for playback. The player used by this site is our LCEVC-enabled reference integration in Shaka Player.

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/O5YitEMoTg7ZYr7r1V4z/image.png)

### 2. V-Nova Windows App

As mentioned above, V-Nova's demo application on Windows can play back local files as well as remote content and can be [downloaded here from the Microsoft Store](https://www.microsoft.com/en-qa/p/v-nova/9n9c2z21xhl9?cid=msft_web_chart\&activetab=pivot:overviewtab).

To play back any content simply select "Launch Player" from the app's login screen:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/wkDxOQfbi6wjZ91byjNb/image.png)

Once the player has launched, specify the URL you wish to play and click "Load":

![V-Nova Windows app toolbar](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/6p5j0dDU5lVKJYOXmyDi/image.png)

### 3. Using other reference integrations

If as part of your evaluation you have been provided downloads of any of the following, the included test applications or reference builds can be used to play back any streams of your choosing:

* **AndroidX (ExoPlayer)** - the included Exo-DIL APK can be installed and used to play remote streams
* **Shaka Player** - Simply host the reference demo build on a web server and specify your streams in the URL field at the top of the player.
* **hls.js** - Simply host the reference demo build on a web server and specify your streams in the URL field at the top of the player.
* **LCEVC AVPlayer** - the provided github project contains a test application which can be built and used to testing of streams on iOS devices.

## Playing back content encoded on the V-Nova platform

Playing back content that you have encoded on the V-Nova platform is very straightforward as the platform is fully integrated with the V-Nova demo applications which are available from all of the major app stores. Simply sign into the V-Nova demo application using your V-Nova Platform credentials and all of your content will be listed for playback directly within the app.

Search "V-Nova" or "LCEVC" on the relevant store or follow the links below:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/dCQshMY9I43Hys391qen/image.png)

[Download here from the Apple AppStore](https://apps.apple.com/gb/app/v-nova/id1363898377)

[Download here from the Google PlayStore](https://play.google.com/store/apps/details?id=com.vnova.PERSEUS\&hl=en_GB\&gl=US)

[Download here from the Microsoft Store](https://www.microsoft.com/en-qa/p/v-nova/9n9c2z21xhl9?cid=msft_web_chart\&activetab=pivot:overviewtab)

[A website version of the V-Nova application is also available here.](https://player.v-nova.com)

## Integrating playback in your applications

LCEVC-enhanced decoding is deployable in software at a variety of levels in your software stack. After testing encoding and playback, should you wish to proceed with testing integration within your own players please review the documentation sections on our [Reference Integrations](/technologies/mpeg-5.lcevc/integrations/integrations) and Decoder Integration Layer and [contact us](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/) for more details.

A range of other decoding implementations are also available including AOSP and MFT. [Contact us](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/) to find out more.


# SDK


# LCEVC Decoder for Web (LCEVCdecJS)

LCEVC Decoder for Web (LCEVCdecJS)

## Introduction

LCEVCdecJS is V-Nova's implementation of the LCEVC (MPEG-5 Part 2) decoder for web-based playback.&#x20;

LCEVCdecJS is [available on GitHub](https://github.com/v-novaltd/LCEVCdecJS) as a 'source available' repository.&#x20;

This section provides an overview of the LCEVC decoding components for Web, included as part of the V-Nova LCEVC SDK, and provides instructions on how to integrate them into different HTML5 players.

### LCEVC-enhanced HTML5 pipeline

![Figure 1: HTML5 decoding with and without LCEVC enhancement](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/MUd3MTUJmQ2x4M3LGQu4/lcevcdecjs-pipeline-diagram.png)

As shown in Figure 1, the main change in the video pipeline due to enabling LCEVC decoding is to hide the original video tag region and then play back video on a new canvas region. This new canvas region and passing the MSE SourceBuffers into the LCEVCdecJS library are the main requirements of the modified video pipeline. The following sections provide further detail on these aspects as well as integration with web based video players.

The HTML5 decoding takes the following inputs:

1. Video segment as an MSE SourceBuffer that is also fed to the HTML Video Element;
2. Video element for the base decoded video picture;
3. A div element as destination for subtitles;
4. Canvas element as destination for the LCEVC-enhanced decoded video.

By using the requestAnimationFrame API, the LCEVC-enhanced decoder extracts each base video frame from the video element and renders the LCEVC-enhanced output frame on the given canvas element.

Since subtitles (e.g WebVTT) are typically rendered by the browser on the video tag, there is currently no way for the LCEVC decoder to extract that region and move it to the canvas. As a result, the solution currently offered by the decoder is to render the cues via simple, yet customizable, div rendering on top of the canvas element.

### Supported browsers and platforms

The following capabilities are required in order to integrate V-Nova LCEVC HTML5 playback:

* WebGL 1.0 with FrameBuffer capability. This is almost universal among current browsers and systems.
* WebWorkers / WebAssembly.
* WebAudio with Delay node functionality.

Typical steps taken to verify browser type and feature required including example code are described below:

**Step 1:** Detect the supported browser and browser version.

The supported browser and versions for different operating systems:

{% tabs %}
{% tab title="Windows" %}
The supported browser and versions for Windows:

* Chrome (version ≥ 57)
* Firefox (version ≥ 52)
* Edge (version ≥ 16)
  {% endtab %}

{% tab title="MacOS" %}
The supported browser and versions for MacOS:

* Chrome (version ≥ 57)
* Firefox (version ≥ 52)
* Safari (version ≥ 11)
  {% endtab %}

{% tab title="Ubuntu" %}
The supported browser and versions for Ubuntu 18.04 LTS:

* Chrome version 92.0.4515.131&#x20;
* Firefox version 94.0

Specified or newer versions are supported.
{% endtab %}
{% endtabs %}

Sample code for detecting the current browser and browser version:

<details>

<summary>Sample Code: Detect the current browser and browser version</summary>

```javascript
var nVer = navigator.appVersion;
var nAgt = navigator.userAgent;
var browserName = navigator.appName;
var fullVersion, verOffset, majorVersion;

// In Chrome, the true version is after "Chrome".
if ((verOffset = nAgt.indexOf('Chrome')) !== -1)
{
	browserName = 'Chrome';
	fullVersion = nAgt.substring(verOffset + 7);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Safari, the true version is after "Safari" or after "Version".
else if ((verOffset = nAgt.indexOf('Safari')) !== -1)
{
	browserName = 'Safari';
	fullVersion = nAgt.substring(verOffset + 7);
	if ((verOffset = nAgt.indexOf('Version')) !== -1)
		fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
// In Firefox, the true version is after "Firefox".
else if ((verOffset = nAgt.indexOf('Firefox')) != -1)
{
	browserName = 'Firefox';
	fullVersion = nAgt.substring(verOffset + 8);
	majorVersion = parseInt('' + fullVersion, 10);
}
//Edge
else if ((verOffset = nAgt.indexOf('Edge')) != -1)
{
	browserName = 'Edge';
	fullVersion = nAgt.substring(verOffset + 5);
	majorVersion = parseInt('' + fullVersion, 10);
}
else
{
	//Browser not supported.
}
```

</details>

**Step 2:** Detect the OS.

It is advised that the supported browser for the respective OS is used, as specified in Step 1.&#x20;

<details>

<summary>Sample Code: Detect the operating system</summary>

```javascript
var OSName;
if (navigator.appVersion.indexOf('Win') != -1) OSName = 'Windows';
if (navigator.appVersion.indexOf('Mac') != -1) OSName = 'MacOS';
if (navigator.appVersion.indexOf('X11') != -1) OSName = 'Unix';
if (navigator.appVersion.indexOf('Linux') != -1) OSName = 'Linux';
```

</details>

**Step 3:** Check that WebGL 1.0 is available.

Create a canvas element. Note that the canvas is not added to the document itself, so it is never displayed in the browser window.

<details>

<summary>Sample Code: Check for WebGL support</summary>

```javascript
var canvas = document.createElement('canvas');

// Get WebGLRenderingContext from canvas element.
var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');

if(gl && gl instanceof WebGLRenderingContext) {
	// Congratulations! Your browser supports WebGL.
} else {
	// Your browser may not support WebGL.
}
```

</details>

**Step 4:** Check if the browser supports/does not support WebWorkers.

<details>

<summary>Sample Code: Check for WebWorker support</summary>

```javascript
if (window.Worker) {
	// Congratulations! Your browser supports WebWorkers.”
} else {
	// Your browser does not support WebWorkers.
}
```

</details>

**Step 5:** If all of the preceding requirements(steps) are satisfied, then proceed to the video playback. Otherwise, display an alert message to the user.

### Components

The HTML5 delivery for the LCEVC SDK consists of the following sets of Javascript and Web Assembly files specific to decoding LCEVC-enhanced streams:

* liblcevc\_dpi.wasm: V-Nova LCEVC WebAssembly decoder library.
* lcevc\_dec.min.js: V-Nova LCEVC JavaScript player.
* lcevc\_dec\_patch\_*xyz*.js: V-Nova LCEVC JavaScript functions that are used in conjunction with the specific player being used e.g. lcevc\_dec\_patch\_hls.js

## LCEVCdecJS integration guide

This section outlines the steps and considerations needed to integrate LCEVCdecJS with any HTML5 player based on Media Source Extensions (MSE), and how to write your own patch file `lcevc_dec_patch_xyz.js`. The examples given here refer to Hls.js but the principles are consistent for other players.

### Overview

The general concept is that the base layer of the video (the image of the base frame) is provided by the browser's native video tag, while the LCEVC enhancement data is provided by extra data stored in the video stream. The supplied LCEVC library decodes these enhancements into a texture, then using WebGL, composites this texture on top of the original video image to create the final enhanced frame.

It is essential that the timestamp of the metadata exactly matches the timestamp of the video image, which can be achieved using JavaScript, as they are provided in different ways by the browser.

Each video player is coded differently. With some (such as Hls.js) it may be possible just to hotwire global JavaScript properties and functions, and provide the integration through a patch script. With others the only option may be to compile from source, and bundle an entire new version of the library.

The steps taken in **Hls.js** serve as an example and starting point to adding LCEVC support to the video technology you wish to target using a patch file. As you read through this documentation, make sure to refer to the methods outlined below, which are crucial for invoking LCEVC-enhancement decoding. Identify similar functionalities within the player you are using and apply similar modifications to ensure smooth integration with LCEVC capabilities.

### 1. Initialize and Check Dependencies&#x20;

The initialization block sets up the environment by checking for the presence of two important dependencies, `Hls.js` and `LCEVCdec` within the window object. If either of these dependencies is not found, an error is thrown to ensure that the necessary prerequisites for the subsequent code execution are met.

```javascript
const { Hls } = window;
if (!Hls) {
  throw new TypeError('Hls.js needs to be loaded before patch can be applied.');
}

const { LCEVCdec } = window;
if (!LCEVCdec) {
  throw new TypeError('LCEVC needs to be loaded before patch can be applied.');
}

```

### 2. Opening and Closing LCEVC

The recommended way to initialise the LCEVC object is through an `attachLcevc` function that should be defined on the player object and called directly after the player has been created. We will demonstrate how to define the `attachLcevc` function in further sections. This is known as creating a patch file and patching the player.

```javascript
// Get reference to video that LCEVCdecJS will use for input
video = document.getElementById('video');

// Get reference to canvas that LCEVCdecJS will use for rendering
canvas = document.getElementById('canvas');

// Set LCEVC config options
const lcevcDecConfig = {
  debugStats: true,
  playerControls: {
    playerID: 'player-controls',
    enabled: true,
    position: 'top',
  }
};

// Create LCEVCdecJS instance within Hls.js object
hls.attachLcevc(canvas, lcevcDecConfig);
```

A canvas reference is passed to `attachLcevc` (enhanced frames will be rendered to this canvas using WebGL) along with the configuration options (some of which **can** **not** be modified after creation).

Moving on to the integration methods for Hls.js and LCEVCdecJS, these functions establish the necessary connections and event listeners to ensure smooth interaction between the HLS player and the LCEVC decoder. They handle critical events such as media attaching, manifest loading, and buffer management to synchronize the behavior of both components effectively.

The function `attachLcevc` could be defined in the patch file as an extension of the **Hls.js** prototype. By listening to the `MEDIA_ATTACHING` and `MEDIA_DETACHING` events, the player is able to create an instance of the LCEVCdecJS object in memory and remove it from memory when the player closes, seamlessly, without the host page needing any extra callbacks or scripts.

```javascript
// attachLcevc connects the instanced Hls.js player to the LCEVC decoder
Hls.prototype.attachLcevc = function (canvas, config) {
  this.canvas = canvas;
  this.LCEVCdecConfig = config;

  // attachMedia connects the new HLS player to the V-Nova's
  // LCEVC decoder and passes the objects to it.
  
  // MEDIA_ATTACHING instances the LCEVC decoder, passing the video tag,
  // the canvas tag, and the LCEVC config
    this.on(Hls.Events.MEDIA_ATTACHING, function mediaAttaching(_, data) {
    this.LCEVCdec = new LCEVCdec.LCEVCdec(data.media, this.canvas, this.LCEVCdecConfig);
  });

  // MEDIA_DETACHING removes the LCEVC components from memory
  this.on(Hls.Events.MEDIA_DETACHING, (event) => {
    if (this.LCEVCdec) {
      this.LCEVCdec.close();
      this.LCEVCdec = null;
    }
  });

  // ...
  // Further methods for appending buffer data, handling profile
  // switching events, etc, should be placed here within the
  // same attachLcevc function.
}
```

### 3. Appending Buffer Data

The main job of any video stream player is to parse the manifest file (.m3u8/.mpd) and manage the network, so that the correct video byte data is downloaded from the server and attached to the video player at the right time.&#x20;

Each of the video technologies mentioned; e.g. Hls.js contains a function or a callback which appends incoming byte data to a MediaSourceBuffer object. At this point video data should be intercepted and a copy of the data also passed to the LCEVC decoder.

For versions of Hls.js later than `0.14.17`, the code sample below listens to buffer appending events. When video or audio/video data is appended to the buffer, it forwards this data to the LCEVC decoder for processing.

<pre class="language-javascript"><code class="lang-javascript"><strong>this.on(Hls.Events.BUFFER_APPENDING, function bufferFlushing(_, data) {
</strong>  if ((data.type === 'video' || data.type === 'audiovideo') &#x26;&#x26; this.LCEVCdec) {
    this.LCEVCdec.appendBuffer(
      data.data,
      data.type,
      data.frag.level
    );
  }
});
</code></pre>

The LCEVCdecJS function `LCEVCdec.appendBuffer` passes the `arrayBuffer` to a WebWorker which demuxes and decodes it, then stores the extracted LCEVC data.

### 4. Maintaining the Profile Level and PTS Drift

An adaptive bitrate video stream can be composed of video segments with different bitrates (to accommodate changing network conditions) that are concatenated to create a seamless playback inside a single video tag. Because these segments might have slightly different start and end timestamps it is possible for the residual metadata to become misaligned with the base video layer.

The LCEVCdecJS `appendBuffer` function contains a parameter for supplying a `timestampOffset`, which may sometimes be supplied by the player to indicate any applied time offsets to the base video. This is helpful when the player provides different time offsets per each profile, which LCEVCdecJS is equipped to handle using `appendBuffer(data, type, level, timestampOffset)`. Additionally, PTS drift is also available for cases where `timestampOffset` is not sufficient, which is discussed in this section.&#x20;

Some players will have an event or a callback which will indicate that the PTS offset has changed when parsing a new fragment. Sometimes they come with extra information like type of fragment, level and drift time. This extra data helps LCEVC with the synchronisation of the enhancement layer.

In Hls.js the `LEVEL_PTS_UPDATED` event is used:

```javascript
// Add level PTS update event to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_PTS_UPDATED, function levelPTSUpdated(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.newPts(data.start, data.end, data.level, data.type, data.drift);
  }
});
```

Whenever the player indicates that the network has changed, the settings of the video profile (collected from metadata) must be flushed. This happens whenever a level switch occurs, or the player changes video source.

Event handlers related to level switching (`LEVEL_SWITCHING`, `LEVEL_SWITCHED`)  ensure that the LCEVC decoder receives updated information about the current playback level and any level switches. This allows the LCEVC decoder to adjust its processing parameters accordingly, optimizing the decoding process for different quality levels.

LCEVCdecJS also functions differently depending on the container format (i.e. `TS`, `MP4`, `WEBM`) and as such it requires the container format to be set using `setContainerFormat(...)`. Good place to do this is either on manifest load, or in the level switched event.

In Hls.js the following events are used:

```javascript
// Buffer flushing events clears residuals when buffer flushed.
this.on(Hls.Events.BUFFER_FLUSHING, function bufferFlushing(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.flushBuffer(data.startOffset, data.endOffset);
  }
});

// Send level switching event data level to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_SWITCHING, function levelSwitching(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.setLevelSwitching(data.level);
  }
});

// Send level switched event data level to ensure the correct functionality
// of V-Nova LCEVC.
this.on(Hls.Events.LEVEL_SWITCHED, function levelSwitchted(_, data) {
  if (this.LCEVCdec) {
    this.LCEVCdec.setCurrentLevel(data.level);

    if (this.streamController
      && this.streamController.fragCurrent
      && this.streamController.fragCurrent.initSegment
      && this.streamController.fragCurrent.relurl) {
      if (this.streamController.fragCurrent.relurl.includes('.mp4')) {
        this.LCEVCdec.setContainerFormat(2);
      }
    }
  }
});

```

Container format will likely be signalled differently by different players. In the code snippet below, the container format is checked against various formats (TS, WEBM, MP4), and based on the match, the corresponding container format index is passed to `setContainerFormat(...)` to configure LCEVCdecJS accordingly. This ensures that the decoder operates optimally based on the specific characteristics of the media container.

```javascript
if (containerFormat.includes('video/ts')) {
    this.LCEVCdec.setContainerFormat(0);
} else if (containerFormat.includes('video/webm')) {
    this.LCEVCdec.setContainerFormat(1);
} else if (containerFormat.includes('video/mp4')) {
    this.LCEVCdec.setContainerFormat(2);
}
```

## API

The interface between the webpage script and the LCEVC-enhanced player is maintained by a simple API using the following properties, methods, and events.

### Configurations options

Configurations parameters can be provided to lcevc\_dec.min.js to create an `lcevcDec` object.

```javascript
const LCEVCdecDefaultConfig = {
  debugStats: false,
  renderAtDisplaySize: true,
  logLevel: 0,
  dps: true,
  dynamicPerformanceScaling: true,
  iOSFallback: false,
  logo: true,
  drawLogo: true,
  loop: false,
  poster: true,
  ebmlVersion: '3.0.0',
  mp4boxVersion: '0.5.2',
  ebmlLoadOnStartup: false,
  mp4boxLoadOnStartup: true,

  playerControls: {
    controlsID: 'player-controls',
    enabled: false,
    playButton: true,
    seekButton: false,
    volumeButton: false,
    fullscreenButton: true,
    fullscreenButtonPos: 'left',
    toggleButton: false,
    toggleButtonPos: 'right',
    position: 'bottom',
    offsetY: 24,
    offsetX: 0,
    colorPlayed: 0x5e7ebd,
    colorBuffered: 0xadadad,
    colorUnplayed: 0x5e5e5e,
    fullscreenElement: null
  }
};
```

#### debugStats

Show debug statistics including framerate, dropped frame, and other statistics as an overlay on top of the main canvas.

Default: `false`

#### renderAtDisplaySize

Use the display size instead of the video size when rendering. Useful when the display size is lower than the video size. For example, if the video is 1080p but the player size is 720p, it will render at 720p.

Default: `True`

#### logLevel

Set the LCEVC log level. Depending on the level specified, different levels of message feedback detail will be printed in the console.

Default: `LCEVCdec.LogLevel.NONE`

#### dps (Dynamic Performance Scaling)

Enable or disable dynamic performance scaling. Dynamic performance scaling monitors playback to ensure that the number of frames being dropped due to constrained system resources hasn't reached an unacceptable level. If the decoding system is struggling, LCEVC decoding will be temporarily paused with the video just displaying the base video decode.

Default: `True`

#### dynamicPerformanceScaling

See **dps**.

#### playerControls

Enables a simple player controller with some customisation options. It is used to debug the actions of a player controller. Also, it gives some information about the stream like video size, selected profile, frame rate and timestamp. It can be configured with the following parameters:

* enabled: `True` to enable it, otherwise `False`.
* position: CSS style position of the controls.
* offsetY: Vertical position offset of the controls.
* offsetX: Horizontal position offset of the controls.
* colorPlayed: Hex color of the played time bar.
* colorBuffered: Hex color of the buffered video.
* fullscreenElement: `HTMLElement` of the player.

Default: `null`

### Version control

#### LCEVCdec.LCEVC\_DEC\_BUILD\_DATE

Returns the build date.

### Log level API

#### LCEVCdec.LogLevel

Log level enumeration:

* NONE: 0.
* ERROR: 1.
* WARNING: 2.
* INFO: 3.
* DEBUG: 4.
* VERBOSE: 5.

### LCEVCdec API

#### new LCEVCdec.LCEVCdec(video, canvas, configOptions)

Construct a new **LCEVC** decoder object using the `HTMLVideoElement` as the base video element, `HTMLCanvasElement` as the display canvas, and the LCEVC configuration options.

### LCEVCdec instance methods

#### lcevcDec.video get

Return the `HTMLVideoElement` used as the base video element.

#### lcevcDec.isFullscreen get

Return `True` if the video is fullscreen, otherwise `False`.

#### lcevcDec.isLive get

Return `True` if the video is live, otherwise `False`.

#### lcevcDec.isPerformanceScalingEnabled get

Return `True` if dynamic performance scaling is enabled, otherwise `False`.

#### lcevcDec.enablePerformanceScaling(value)

Enable or disable the dynamic performance scaling.

#### lcevcDec.isPerformanceScalingActive get

Return `True` if the performance scaling is active, otherwise `False`.

#### lcevcDec.isLcevcEnabled get

Return `True` if LCEVC decoding is enabled, otherwise `False`.

#### lcevcDec.lcevcDataDetected get

Return `True` if LCEVC data is detected in the video, otherwise `False`.

#### lcevcDec.frameWidth get

Return the width size of the displayed frame.

#### lcevcDec.frameHeight get

Return the height size of the displayed frame.

#### lcevcDec.currentLevel get

Return the number value of the current level.

#### lcevcDec.firstLcevcSegmentLoaded get

Return `True` if a segment with LCEVC data has been loaded, otherwise `False`.

#### lcevcDec.aspectRatio get

Returns the value of the aspect ratio of the video.

#### lcevcDec.frameRate get

Returns the frame rate of the video.

#### lcevcDec.getConfigOption(option)

Get the value of the `option` of the LCEVC configuration.

#### lcevcDec.setConfigOption(option, value)

Set the `option` with the `value` to the LCEVC configuration.

#### lcevcDec.logLevel get/set

Get or set the log level to the given `LCEVCdec.LogLevel` (default NONE).

#### lcevcDec.profileIn get

Get the input color profile.

#### lcevcDec.setProfileIn(profileName)

Set the input color profile to the given profile name.

#### lcevcDec.profileOut get

Get the output color profile.

#### lcevcDec.setProfileOut(profileName)

Set the output color profile to the given profile name.

#### lcevcDec.onFullscreen(enable)

Inform the LCEVC decoder that the player has triggered a fullscreen event.

#### lcevcDec.clearTemporal()

Clears the LCEVC temporal buffer. This will remove the current residuals and should only be used when moving or seeking in the video or when changing the profile.

#### lcevcDec.setCurrentLevel(level)

Notifies LCEVC decoder that a level loaded event has been triggered passing the new `level`. This ensures that LCEVC data is managed correctly upon loading a new level.

#### lcevcDec.setLevelSwitching(level)

Notifies LCEVC decoder that a level switching event has been triggered passing the new `level`. This ensures that LCEVC data is managed correctly on the transition between levels.

#### lcevcDec.appendBuffer(data, type, level, timestampOffset)

Integration function only.

Receives video buffer data and sends it to the worker to detect and extract LCEVC data. The arguments are:

* data: The video buffer data.
* type: The type of the data, can be `'video'`, `'audio'` or `'audiovideo'`.
* level: The current ABR profile/rendition/level ID of the data.
* timestampOffset: offset added to the timestamps in the fragment (defaults to 0.0).

#### lcevcDec.flushBuffer(startTime, endTime)

Flush the residual and drift data of the given interval of time in milliseconds.

#### lcevcDec.resetBuffer()

Reset the internal buffers. This should only be used when changing the video. Calling this during  playback may lead to unexpected behaviour.

#### lcevcDec.newPts(start, end, type, level, drift)

Integration function only.

Instructs LCEVC decoder that new information has been updated after parsing a fragment.&#x20;

The type, level and drift are optional, but providing them helps LCEVC decoder to maintain correct synchronisation between the base video and decoded enhancement layers.

* start: The start time in milliseconds of the fragment.
* end: The end time in milliseconds of the fragment.
* type: Type of the fragment, can be `'video'` or `'audio'`.
* level: The level of the parsed fragment.
* drift: The drift amount of time.

#### lcevcDec.setContainerFormat(containerFormat)

Set the container flag, whether the container is `TS`, `WEBM`, or `MP4`. LCEVCdecJS functions differently depending on the container format. Good place to do this is either on manifest load, or in the level switched event of a player.

* 0: signals `TS` container to LCEVCdecJS
* 1: signals `WEBM` container to LCEVCdecJS
* 2: signals  `MP4` container to LCEVCdecJS

#### lcevcDec.displayError(message)

Prints an error message in the canvas where the video is displayed.

#### lcevcDec.clearError()

Clears the error message of the canvas if any.

#### lcevcDec.close()

Closes the LCEVC decoder and releases memory in use. This should only be used when changing the video. Calling this during playback may lead to unexpected behaviour.

### Events API

#### lcevcDec.on(type, listener)

Attach an event listener of `LCEVCdec.Events` type to the LCEVC decoder.

#### lcevcDec.off(type, listener)

Remove the event listener of the given `LCEVCdec.Events` type.

### Runtime events

#### LCEVCdec.Events.PERFORMANCE\_DROPPED

Occurs when Dynamic Performance Scaling is enabled, and GPU system resources are heavily constrained.&#x20;

```javascript
lcevcDec.on(LCEVCdec.Events.PERFORMANCE_DROPPED, function(){
    console.log('Performance Dropped');
}
```

#### LCEVCdec.Events.PERFORMANCE\_RESTORED

Occurs after a PERFORMANCE\_DROPPED event, when the GPU is able to run at optimal speed again. LCEVC enhancement decoding has been restored.

```javascript
lcevcDec.on(LCEVCdec.Events.PERFORMANCE_RESTORED, function(){
    console.log('Performance Restored');
}
```


# Decoder Integration Layer (DIL)

## Introduction

The DIL (Decoder Integration Layer) is the V-Nova library used to integrate LCEVC decoding and rendering to an existing legacy decoder implementation.

The DIL can be used in both:

* Application integration - the DIL renders directly onto an on-screen surface, provided by the client, of arbitrary size (generally different from the content resolution). Examples: ExoPlayer (Android).
* OS Integration - the DIL decodes onto a buffer or draws onto an off screen texture of the same size of the content final resolution, it doesn’t handle the final render to display, including YUV to RGB conversion, and resizing to the destination surface. Examples: MFT (Windows), OMX (Android).

The following block diagram illustrates the role of the DIL more clearly. In conjunction with a base decoder, typically provided by the OS, it offers a complete solution from buffer to output.

![Figure 1 Unmodified and LCEVC Pipeline using the DIL  ](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/dR1Kc5RKjRsYaoYL7h3S/Capture.PNG)

##

## DIL Types and Enumerations

### DIL\_NALFormat enum

DIL\_NALFormat represents the two possible NAL Unit formats, for each of the MPEG base codecs the DIL currently supports: H264 (ISO/IEC 14496-10, aka AVC) and H265 (ISO/IEC 23008-2, aka HEVC) NAL, where Annex B indicates 3 byte 0x000001 or 4 byte 0x00000001 start code prefix as described in Annex B of (H264 or HEVC), LP (Length Prefix) indicates a 4 byte prefix with the length of the NAL unit as MSBF 32 bit unsigned int.&#x20;

```cpp
typedef enum DIL_NALFormat
{
    DIL_UnknownNALFormat = 0,
    DIL_H264_AnnexB,
    DIL_H265_AnnexB,
    DIL_H264_LP,
    DIL_H265_LP
} DIL_NALFormat;
```

### DIL\_ReturnCode enum

DIL\_ReturnCode represents the possible API return codes. Most of the API calls will return one of these values.

```cpp
typedef enum DIL_ReturnCode
{
    DIL_RC_Success      =  0,       /* The API call completed successfully */
    DIL_RC_Error        = -1,       /* A generic catch all error */
    DIL_RC_InvalidParam = -2,       /* The user supplied an invalid parameter to the function call */
    DIL_RC_NotFound     = -3,       /* The query call failed to find the item by name, but didn't not generate an error. */
    DIL_RC_NotSupported = -4,       /* The functionality requested is not supported on the running system. */
    DIL_RC_Flushed      = -5,       /* The requested operation failed because it was flushed via DIL_Flush. */
    DIL_RC_Timeout      = -6        /* The requested operation failed because it timedout. */
} DIL_ReturnCode;
```

### DIL\_ColorFormat enum

This enum represents the supported colour formats, currently only 8 bit per sample formats are supported For a detailed description of the formats see: <https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html>

```cpp
typedef enum DIL_ColorFormat
{
    DIL_UnknownColorFormat = 0,     /* Unspecified colour format */
    DIL_I420_8,                     /* 8 bit 4:2:0 YUV Planar colour format: Y, U and V on separate planes */
    DIL_NV12_8,                     /* 8 bit 4:2:0 YUV SemiPlanar colour format: Y plane and UV interleaved plane */
    DIL_RGB_8,                      /* 8 bit Interleaved R, G, B planes 24 bit per sample */
    DIL_RGBA_8,                     /* 8 bit Interleaved R, G, B and A planes 32 bit per sample */
    DIL_I420_10LE,                  /* 10 bit Little Endian 4:2:0 YUV Planar colour format: Y, U and V on separate planes */

} DIL_ColorFormat;

```

### DIL\_ImageType enum

This enum identifies the type of planes the DIL\_Image consists of; planes are non-overlapping sets of samples belonging to one or more colour channels with a constant sample stride, so RGB is a single plane, because R G and B pointed at separately would overlap UV from NV12 is a single plane, because U and V pointed at separately would overlap Y U and V from I420 are three planes because they do not overlap

```cpp
typedef enum DIL_ImageType
{
    DIL_UnknownImageType = 0,       /* Unspecified image type */
    DIL_Buffer_Array,               /* One buffer per plane, even if image is on a single memory allocation, an array of planes shall be set by assigning the right pointers to plane buffers. DIL allocated buffers have contiguous memory allocation */
    DIL_Texture_Array,              /* One texture per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
    DIL_HardwareBuffer_Array        /* One HardwareBuffer per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
} DIL_ImageType;

```

### DIL\_ImageDesc struct

Holds configurations of a single DIL\_Image instance

```cpp
typedef struct DIL_ImageDesc
{
    bool            is_managed;     /* True means that Image (and its memory) is managed by DIL  */
    DIL_ImageType   image_type;     /* Whether it is a buffer array or a texture array */
    DIL_ColorFormat color_format;   /* Colour format */
	  uint32_t        width;          /* Nominal net width of the image in luma samples, i.e. no alignment to macroblocks or striding */
    uint32_t        height;         /* Nominal net height of the image in luma samples, i.e. no sliceheight or padding */
    bool            can_modify;     /* Specifying wether content of image is readOnly or changable */
    bool            can_resize;     /* Specifying wether allocated memory is resizable or not. (Does not apply to DIL_Buffer_External) */
    uint32_t        options;        /* options used in creating the Image, in the case of HardwareBuffers this is a combination of  DIL_HardwareBufferUsageOption */
} DIL_ImageDesc;
```

### DIL\_ContextSettings struct

This structure holds parameters related to the window surface the DIL would render into, in case of on screen rendering those parameters are normally provided by the windowing system, e.g. EGL, GLFW.

```cpp
typedef struct DIL_ContextSettings
{
    void* gl_external_window;       /* Surface window, from the windowing system */
    void* gl_external_context;      /* Context, from the windowing system */
    bool  is_secure;                /* True if the client has initialised window and context as secure for protected content */
} DIL_ContextSettings;

```

### DIL\_RenderInformation struct

This structure captures the properties related to the (on screen) window rendering, they do not apply for buffer or texture output (off screen). NOTE: rotation is a per frame property because it may change frame by frame due to for ex. the encoding device rotating while shooting.

```cpp
typedef struct DIL_RenderInformation
{
    uint32_t rotation;                  /* Degrees of rotation for the final rendering on the destination window, in 90 unit increments clockwise */
    float    pixel_aspect_ratio_num;    /* Pixel Aspect Ratio of width to height, this is the numerator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
    float    pixel_aspect_ratio_den;    /* Pixel Aspect Ratio of width to height, this is the denominator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
} DIL_RenderInformation;
```

### DIL\_DecodeInformation struct

This structure captures properties related to the decoding process, width and height of the output, whether LCEVC data has been found and whether it has been applied.

```cpp
typedef struct DIL_DecodeInformation
{
    uint32_t width;                     /* width of the decoded frame */
    uint32_t height;                    /* height of the decoded frame */
    bool     skip_requested;            /* decode skip was requested */
    bool     lcevc_enhanced;            /* true if the frame has been enhanced by lcevc decoding */
    bool     lcevc_available;           /* true if lcevc encoded data is available for this frame */
} DIL_DecodeInformation;
```

### DIL\_ImagePlaneBufferDesc struct

This struct describes a plane memory buffer

```cpp
typedef struct DIL_ImagePlaneBufferDesc
{
    uint8_t* data;                      /* Pointer to first byte of the first active sample of the plane buffer, NOTE: active means no cropping, the active origin is 0,0 */
    uint32_t sample_byte_stride;        /* Distance in bytes between consecutive sample values of the same channel, for ex. UV plane on 8bit depth this will be 2 */
    uint32_t row_byte_stride;           /* Distance in bytes between the first sample of two consecutive rows */
} DIL_ImagePlaneBufferDesc;

```

### DIL\_ImagePlaneTextureDesc struct

This struct describes a plane OpenGL texture

```cpp
typedef struct DIL_ImagePlaneTextureDesc
{
    uint32_t target;                /* OpenGL/GLES texture target, as in glBindTexture target */
    uint32_t id;                    /* OpenGL/GLES texture id. Unused indexes should be set to 0 */
    uint32_t width;                 /* OpenGL/GLES texture width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* OpenGL/GLES texture height, which might have extra padding, NOTE: the active height is in the parent Image struct */
    int32_t  internal_format;       /* OpenGL/GLES (GLint) sized internal format as from Table 2 of glTexImage2D man page */
} DIL_ImagePlaneTextureDesc;

```

### DIL\_HardwareBufferUsageOption enum

This enum use used to set the UsageOption when creating HardwareBuffer DIL\_Image types. It should be provided in the "hardware\_buffer\_usage\_option" tag when configuring the DIL object

```cpp
typedef enum DIL_HardwareBufferUsageOption
{
    UsageOption_None = 0,

    UsageOption_Read_Never  = (1 << 0),
    UsageOption_Read_Rarely = (1 << 1),
    UsageOption_Read_Often  = (1 << 2),

    UsageOption_Write_Never  = (1 << 4),
    UsageOption_Write_Rarely = (1 << 5),
    UsageOption_Write_Often  = (1 << 6),

    UsageOption_Protected_Content = (1 << 7),

    UsageOption_GPU_Read          = (1 << 8),
    UsageOption_GPU_Write         = (1 << 9)
} DIL_HardwareBufferUsageOption;

```

### DIL\_ImagePlaneHardwareBufferDesc struct

This struct describes a plane HardwareBuffer

```cpp
typedef struct DIL_ImagePlaneHardwareBufferDesc
{
    uint32_t width;                 /* width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* height, which might have extra padding, NOTE: the active height is in the parent Image struct */
		void*    nativeObject;          /* platform native object/id for the HardwareBuffer */
} DIL_ImagePlaneHardwareBufferDesc;

```

## DIL Properties API

### DIL\_QueryProperty

Obtain a specific named property. This function provides a way of looking up a property by name to observe it's current value, this may be of use for a user to query DIL properties

```cpp
DIL_ReturnCode DIL_QueryProperty(DIL_Decoder decoder, const char* name, DILProperty* output);
```

### DIL\_SetProperty

Set a specific named property. This function provides a way of changing a property by name

```cpp
DIL_ReturnCode DIL_SetProperty(DIL_Decoder decoder, const char* name, DILProperty* value);
```

### DIL\_QueryPropertyGroup

Obtain all the available properties for a named group

```cpp
DIL_ReturnCode DIL_QueryPropertyGroup(DIL_Decoder decoder, const char* name, DILPropertyGroup* output);
```

### DIL\_ReleaseProperty

Release a previously queried property, this function provides a way of releasing any memory allocated by a call to DIL\_QueryProperty. Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleaseProperty(DIL_Decoder decoder, DILProperty* value);
```

### DIL\_ReleasePropertyGroup

Release a previously queried property group, this function provides a way of releasing any memory allocated by a call to DIL\_QueryPropertyGroup Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleasePropertyGroup(DIL_Decoder decoder, DILPropertyGroup* value);
```

### DILProperty struct.

This structure contains all relevant information used to query a property or metadata from the DIL.

```cpp
typedef struct DILProperty
{
  const char*     name;        /* The name of the property.  */
  const char*     description; /* A description of the property. */
  DILPropertyType type;        /* The value type for this property. */
  union
  {
    int8_t      i8;
    int16_t     i16;
    int32_t     i32;
    int64_t     i64;
    uint8_t     u8;
    uint16_t    u16;
    uint32_t    u32;
    uint64_t    u64;
    float       f;
    double      d;
    const char* str;
    const void* vptr;
    const void* data;
		const DILPropertyGroup* pgptr;
  } value;                  /* A union representing the value of the property. The user must read the field that match the type enum value. */

  const uint8_t* blob;      /* Special field containing a block of contiguous memory when the type enum is set to DIL_PT_Blob. This is allowed to be NULL. */
  uint32_t       blob_size; /* When the blob is not NULL then the length in bytes of the memory, otherwise 0. */
	uint64_t       data_size; /* size of the "data" buffer in bytes */
	bool           can_free;  /* flag to show if the pointer values should be free'ed when the DIL_ReleaseProperty is called */
} DILProperty;
```

### DILPropertyGroup struct.

This structure contains a logical grouping of properties, this is a convenience feature to assist with providing a "pretty" command line, or GUI.

All properties are assigned to a group, and properties can only appear in one group.

```cpp
typedef struct DILPropertyGroup_t
{
  const char*  name;           /* The name for this group of properties. */
  const char*  description;    /* A description of the property group. */
  DILProperty* properties;     /* An array of properties for this group. */
  uint32_t     property_count; /* The length of the properties array. */
	bool         can_free;       /* flag to show if the properties pointer should be free'ed when the DIL_ReleaseProperty is called */
} DILPropertyGroup;
```


# Functionality

### Input & output

As shown in the diagram above, the DIL needs two inputs:

* input buffers - the same buffers that are fed to the base decoder in encoding order;
* base decoded frames - produced by the base decoder in presentation order;

and produces an output onto either a

* buffer or
* offscreen texture or
* onscreen surface.

### Colour formats

The DIL can operate on a base decoded picture in the following colour formats:

* YUV420 I420 - planar Y, U and V with 4:2:0 subsampled chroma;
* YUV420 NV12 - semi-planar Y (planar), and UV (interleaved) with 4:2:0 subsampled chroma;
* RGB/RGBA - raster Red Green Blue, i.e. on one plane with interleaved colour components;

### Modes

The DIL can be configured to work with two types of internal pipeline:

* CPU - all the LCEVC stages are performed in CPU, only using SIMD acceleration, by means of the DPI only, the GPU is used only for the possible YUV/RGB conversion;
* GPU - most of the LCEVC stages are performed in GPU suing GL shaders, including YUV/RGB conversions, while the CPU is only used to produce the LCEVC residual planes at both LOQs

### Output type

The DIL offers decoding to either

* on screen
* off screen

On screen refers to a window from the OS window system, currently EGL and GLFW windowing systems are supported.

Off screen refers to an array of buffers or GL textures in memory. The latter case has been designed to allow the client to manage decode and presentation separately, for example to allow buffering/queuing already decoded pictures before display.

### Other configurable features

On creating a DIL instance, the client can configure additional features, such as:

* OpenGL major and minor versions (to force a specific version over the system’s selected one);
* Use of OpenGL ES;
* Use of 8 bit LCEVC residual planes - instead of 16 bit
* Use of Hardware Buffers - (Apple and Android only)
* Enable an on screen UI for stats and live config;
* Enable dumping stats to local storage;
* Enable dumping raw output frames to local storage;

The clients configures the DIL at creation time by passing a JSON string and a pair of window system related parameters for context and destination surface.

## API Overview

Using the DIL is fairly easy, especially if compared to the more basic DPI, since it operates at a relatively high abstraction level. Apart from obvious creation and destruction of a DIL instance (respectively `DIL_Create()` and `DIL_Destroy()`), the main API calls are effectively two:

* Feed input - `DIL_AddNALData()`
* Decode - `DIL_DecodeandRender()` (for decoding on screen, alternatively `DIL_Decode()` for decoding off screen followed by `DIL_Render()`)
* Feed input - `DIL_AddNALData()`
* Decode - `DIL_DecodeandRender()` (for decoding on screen, alternatively `DIL_Decode()` for decoding off screen followed by `DIL_Render()`)

The DIL also offers an API for the client to retrieve the size of the output picture, which can be useful to pre-allocate buffers or textures of the right size before decoding:

* Get size information - `DIL_GetDecodeInformation()`

Note: input buffers must have NAL byte stream format with either Annex B or Length Prefix format.

### Drop/Skip frames

LCEVC content may be encoded with a temporal feature that requires each and every frame to be actioned on in order to keep the DPI internal temporal reference data correct. As a result when the client skips frames, for ex. because of a seek in the timeline, or drops frames because they are “late”, it shall let the DIL know by calling the following function:

* `DIL_DecodeSkip()`

The DIL does know whether the temporal feature is on and therefore can fall back to a no operation case if that is the case. It is not advised that the client tries to implement this behaviour on its side.

### Synchronous and Asynchronous modes

The DIL can be used in both synchronous or asynchronous mode. This is applicable to the decode and render functions:

* `DIL_Decode()`
* `DIL_DecodeAndRender()`
* `DIL_DecodeSkip()`
* `DIL_Render()`

If the client has previously set the callback function by means of the `DIL_SetDecodedAsyncCallback()` or `DIL_SetRenderAsyncCallback()` API calls, the above decode functions will work asynchronously, therefore returning immediately and triggering the client’s callback function when the decode (or render) has completed. If the callback has not been set the functions will work synchronously.

### Base pass-through mode

The DIL also supports a pass-through mode, in which the base is simply copied on to the output without applying the LCEVC enhancement. This mode is triggered when no LCEVC data is found for the frame and can be set as forced behaviour, for every frame, from the JSON configuration.

## Example Integration Code

In basic terms, a sample integration code using the DIL can be briefly described, among the various phases, as follows:

```cpp
// Include DIL header
#include <lcevc_dil.h>
// Initialisation phase ///////////////////////////////////////////////////////
// Create an instance of the DIL
DIL_ContextSettings context_settings = { 0 };
char * json_settings = "{\
 \"gl_decode\": true,\
 \"gl_es\": true,\
 \"gl_major\": 3,\
 \"gl_minor\": 1,\
 \"swap_interval\": 0,\
 \"fullscreen\": false,\
 \"hardware_buffers\": false,\
 \"use_u8_surface\": false,\
 \"force_passthrough\": false\
}";
// Create a DIL instance
DIL_Decoder decoder = nullptr;
if (DIL_Create(json_settings, &context_settings , &decoder) != DIL_RC_Success)
{
    fprintf(stderr, "Unable to create DIL instance\n");
    exit(-1);
}
// Destruction phase //////////////////////////////////////////////////////////
DIL_Destroy(decoder);
decoder = nullptr;
// Handling input buffers /////////////////////////////////////////////////////
//...
DIL_NALFormat nalFormat;
// Set NAL format, from metadata, properties or parsing the input buffers
// ...
if (DIL_AddNALData(decoder, inputCc, presentationTimeStamp, buffer, length, nalFormat) != DIL_RC_Success)
{
    fprintf(stderr, "Unable to pass input buffer for inputCc=%u pts=%PRId64\n", inputCc, presentationTimeStamp);
    exit(-2);
}
// Handling decoded frame from base decoder ///////////////////////////////////
// ...
// Get size of ouput picture if needed
DIL_DecodeInformation decodeInfo;
if (DIL_GetDecodeInformation(decoder, inputCc, presentationTimeStamp, baseWidth, baseHeight, &decodeInfo) != DIL_RC_Success)
{
    fprintf(stderr, "DIL get decode info failed for inputCc=%u pts=%PRId64\n", inputCc, baseImage.pts;
    exit(-3);
}
// Using decode info
// ...
DIL_Image baseImage;
// Fill the base "Image" struct with the data properties of the base frame
// ...
DIL_RenderInformation renderInfo;
// Fill rotation and pixel aspect ratio
// ...
if (DIL_DecodeAndRender(decoder, inputCc, presentationTimeStamp, baseImage, &renderInfo, &decodeInfo) != DIL_RC_Success)
{
    fprintf(stderr, "DIL decode failed for pts=%PRId64\n", baseImage.pts;
    exit(-4);
}
// Decode info reports if LCEVC is available and has been applied
// ...
```

## DIL API&#x20;

### DIL\_Create

Create an instance of a DIL Decoder

* **json\_settings** JSON string with initialisation parameters
* **context\_settings** settings from the windowing system&#x20;
* **instance decoder** instance created
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Create( const char*                   json_settings,
                           const DIL_ContextSettings*    context_settings,
                           DIL_Decoder                   instance );
```

### DIL\_Destroy

Destroy an instance of a DIL Decoder

* **decoder** instance to be destroyed&#x20;

```cpp
void DIL_Destroy( DIL_Decoder decoder);
```

### DIL\_GetImage

Returns a fresh usable instance of DIL\_Image

* **decoder** DIL instance
* **image\_desc** Describing Image configurations
* **image** Result DIL\_Image if creation is successful, NULL otherwise
* **return** DIL\_RC\_Error if image\_desc contains incompatible values &#x20;

```cpp
DIL_ReturnCode DIL_GetImage(DIL_Decoder decoder, DIL_ImageDesc* image_desc, DIL_Image* image);
```

### **DIL\_ReleaseImage**

Releases instance of DIL\_Image&#x20;

NOTE: No reference calls to image address should be made after this method

* **decoder** DIL instance&#x20;
* **image** DIL\_Image instance to be released&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ReleaseImage(DIL_Decoder decoder, DIL_Image image);
```

### DIL\_ImageGetDesc

Will return DIL\_ImageDesc of the image

* **image** DIL\_Image to query&#x20;
* **desc** Contents of the pointer will be rewritten with image descriptions&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ImageGetDesc(DIL_Image image, DIL_ImageDesc* desc);
```

### DIL\_ImageSetActiveRegion

Will set the active region for an image

* **image**           DIL\_Image instance to modify
* **offset\_x**        X offset for the region
* **offset\_y**        Y offset for the region
* **width**             width for the region
* **height**            height for the region
* **return**            DIL\_RC\_Error if crop region is out of bounds, image\_type doesn't support cropping

```cpp
DIL_ReturnCode DIL_ImageSetActiveRegion( DIL_Image                  image,
                                         uint32_t                   offset_x,
                                         uint32_t                   offset_y,
                                         uint32_t                   width,
                                         uint32_t                   height);
```

### DIL\_ImageGetActiveRegion

Will get the active region for an image

* **image**           DIL\_Image instance to modify
* **offset\_x**        X offset for the region
* **offset\_y**        Y offset for the region
* **width**            width for the region
* **height**           height for the region
* **return**            DIL\_RC\_Error if crop region is out of bounds, image\_type doesn't support cropping

```cpp
DIL_ReturnCode DIL_ImageGetActiveRegion( DIL_Image                  image,
                                         uint32_t*                  offset_x,
                                         uint32_t*                  offset_y,
                                         uint32_t*                  width,
                                         uint32_t*                  height);
```

### DIL\_ImageGetPlanesCount

Will return number image's number of planes

* **image** DIL\_Image to query&#x20;
* **plane\_count** Will be filled with number of planes&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ImageGetPlanesCount(DIL_Image image, uint32_t* plane_count);
```

### DIL\_ImageSetPlaneBuffer

Will set content and configurations of a plane by data buffer

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Buffer or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneBuffer( DIL_Image	                  image,
                                        uint32_t                    plane_index,
                                        DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImageGetPlaneBuffer

Will return content and configurations of a plane

* **image** DIL\_Image to query&#x20;
* **plane\_index** Index of the plane to query&#x20;
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to write to&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_Buffer

```cpp
DIL_ReturnCode DIL_ImageGetPlaneBuffer( DIL_Image                   image,
                                        uint32_t                    plane_index,
                                        DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImageSetPlaneTexture

Will set content and configurations of a plane by OpenGL/GLES texture

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **texture\_desc** DIL\_ImagePlaneTextureDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneTexture( DIL_Image                  image,
                                         uint32_t                   plane_index,
                                         DIL_ImagePlaneTextureDesc* texture_desc);
```

### DIL\_ImageGetPlaneTexture

Will return content and configurations of a plane by OpenGL/GLES texture

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **texture\_desc** DIL\_ImagePlaneTextureDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageGetPlaneTexture( DIL_Image                  image,
                                         uint32_t                   plane_index,
                                         DIL_ImagePlaneTextureDesc* texture_desc);
```

### DIL\_ImageSetPlaneHardwareBuffer

Will set content and configurations of a plane by OS specific Hardware buffer.

* **image**               DIL\_Image instance to modify
* **plane\_index**    Index of the plane to modify
* **desc**                  DIL\_ImagePlaneHardwareBufferDesc instance to read from
* **return**                DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneHardwareBuffer( DIL_Image                         image,
                                                uint32_t                          plane_index,
                                                DIL_ImagePlaneHardwareBufferDesc* desc);
```

### DIL\_ImageGetPlaneHardwareBuffer

Will return content and configurations of a plane by OS specific Hardware buffer.

* **image** DIL\_Image instance to query
* **plane\_index** Index of the plane to query
* **desc** DIL\_ImagePlaneHardwareBufferDesc instance to write to
* **return** Will return DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImageGetPlaneHardwareBuffer( DIL_Image                         image,
                                                uint32_t                          plane_index,
                                                DIL_ImagePlaneHardwareBufferDesc* desc);
```

### DIL\_ImageSetUserData

Will set the user data associated with the image

* **image**  DIL\_Image to set the userdata for
* **userdata**   The userdata to associate with the image
* **return**         DIL\_RC\_InvalidParam if the image, DIL\_RC\_Success otherwise

```cpp
DIL_ReturnCode DIL_ImageSetUserData(DIL_Image image, void* userdata);
```

### DIL\_ImageGetUserData

Will get the user data associated with the image through DIL\_ImageSetUserData

* **image**         DIL\_Image to get the userdata for
* **userdata**    The userdata to associate with the image
* **return**         DIL\_RC\_InvalidParam if the image is invalid or userdata is invalid, DIL\_RC\_Success otherwise

```cpp
DIL_ReturnCode DIL_ImageGetUserData(DIL_Image image, void** userdata);
```

### DIL\_ImagePlaneHardwareBufferLock

Will return content and configurations of a plane buffer obtained by locking the hardware buffer for CPU access

* **image** DIL\_Image to access
* **plane\_index** Index of the plane to query
* **read\_access** Whether we want read access
* **write\_access** Whether we want write access
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to write to
* **return** Will return DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImagePlaneHardwareBufferLock( DIL_Decoder                 decoder,
                                                 DIL_Image                   image,
                                                 uint32_t                    plane_index,
                                                 bool                        read_access,
                                                 bool                        write_access,
                                                 DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImagePlaneHardwareBufferUnlock

Will unlock the hardware buffer plane once CPU access is completed.

* **image** DIL\_Image to unlock
* **plane\_index** Index of the plane to query
* **return** Returns DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImagePlaneHardwareBufferUnlock( DIL_Decoder                 decoder,
                                                   DIL_Image                   image,
                                                   uint32_t                    plane_index);
```

### DIL\_AddTSData

Feed Transport Stream packet(s) for the DIL to extract and store LCEVC data, feed a buffer of TS packets associated with the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **data** pointer to the TS buffer&#x20;
* **size** size of the TS buffer&#x20;
* **returns** DIL API return code

```cpp
DIL_ReturnCode DIL_AddTSData( DIL_Decoder           decoder,
                              uint16_t              input_cc,
                              uint8_t*              data,
                              uint32_t              size );
```

### DIL\_AddWebmData

Feed Webm payload for the DIL to extract and store LCEVC data

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **data** pointer to the TS buffer&#x20;
* **size** size of the TS buffer&#x20;
* **returns** DIL API return code

```cpp
DIL_ReturnCode DIL_AddWebmData( DIL_Decoder         decoder,
                                uint16_t            input_cc,
                                uint8_t*            data,
                                uint32_t            size );
```

### DIL\_AddNALData

Feed NAL unit(s) for the DIL to extract and store LCEVC data, feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed NAL units&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **nalu\_format** format of the NAL units&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_AddNALData( DIL_Decoder          decoder,
                               uint16_t             input_cc,
                               int64_t              timestamp,
                               uint8_t*             data,
                               uint32_t             size,
                               DIL_NALFormat        nalu_format );
```

### DIL\_AddNALDataEx

Feed NAL unit(s) for the DIL to extract and store LCEVC data (Extended), feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp and additionally let the DIL strip the LCEVC payloads and zero pad the trailing bytes, returning the new size before padding&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed NAL units&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **nalu\_format** format of the NAL units
* **return** size of the stripped NAL units buffer

```cpp
uint32_t DIL_AddNALDataEx( DIL_Decoder              decoder,
                           uint16_t                 input_cc,
                           int64_t                  timestamp,
                           uint8_t*                 data,
                           uint32_t                 size,
                           DIL_NALFormat            nalu_format );
```

### DIL\_AddRawData

Feed LCEVC data to the DIL, feed a buffer of pre parsed LCEVC payload data for the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed LCEVC data&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_AddRawData( DIL_Decoder          decoder,
                               uint16_t             input_cc,
                               int64_t              timestamp,
                               uint8_t*             data,
                               uint32_t             size );
```

### DIL\_GetDecodeInformation

Get details about the decoding process for the Access Unit identified by input cc and timestamp

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture&#x20;
* **base\_width** width of the base decoded picture&#x20;
* **base\_height** height of the base decoded picture&#x20;
* **information** pointer to decoder information structure that the DIL will fill&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetDecodeInformation( DIL_Decoder                    decoder,
                                         uint16_t                       input_cc,
                                         int64_t                        timestamp,
                                         uint32_t                       base_width,
                                         uint32_t                       base_height,
                                         DIL_DecodeInformation*         information );
```

### DIL\_Decode

Decode on to an off screen DIL\_Image, get an off screen LCEVC enhanced (or base pass-through) picture from the input base picture relative to input cc and timestamp

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be decoded&#x20;
* **max\_decode\_time\_us maximum** amount of time the decode has to complete, use 0 to disable decode timeout
* **input** pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)&#x20;
* **output** pointer to LCEVC enhanced (or base pass-through) output picture&#x20;
* **decode\_information** pointer to decoder information structure that the DIL will fill (if DIL\_SetDecodedAsyncCallback has not been called)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Decode( DIL_Decoder                      decoder,
                           uint16_t                         input_cc,
                           int64_t                          timestamp,
                           uint32_t                         max_decode_time_us,
                           DIL_Image*                       input,
                           DIL_Image*                       output,
                           DIL_DecodeInformation*           decode_information );
```

### DIL\_Render

Render to back buffer a previously decoded DIL\_Image

NOTE: for the scheduled render to be set DIL\_SetRenderAsyncCallback must have been previously called

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be rendered&#x20;
* **image** pointer to a picture to be rendered to back buffer&#x20;
* **render\_information** pointer to a render information structure&#x20;
* **delay\_us** delay in microseconds for the render to eb executed (0 for no delay)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Render( DIL_Decoder                      decoder,
                           uint16_t                         input_cc,
                           int64_t                          timestamp,
                           DIL_Image                        image,
                           const DIL_RenderInformation*     render_information,
                           int64_t                          delay_us );
```

### DIL\_DecodeAndRender

Decode and render immediately to back buffer

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be decoded and rendered&#x20;
* **max\_decode\_time\_us maximum** amount of time the decode has to complete, use 0 to disable decode timeout
* **input** pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)&#x20;
* **render\_information** pointer to a render information structure&#x20;
* **decode\_information** pointer to decoder information structure that the DIL will fill (if DIL\_SetDecodedAsyncCallback has not been called)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_DecodeAndRender( DIL_Decoder                         decoder,
                                    uint16_t                            input_cc,
                                    int64_t                             timestamp,
                                    uint32_t                            max_decode_time_us,
                                    DIL_Image                           input,
                                    const DIL_RenderInformation*        render_information,
                                    DIL_DecodeInformation*              decode_information );
```

### DIL\_DecodeSkip

Let the DIL know that an Access Unit is not being presented (dropped or skipped), DIL will do the minimum processing to keep internal state consistent for the next picture&#x20;

NOTE: this call will generate a DecodeAsyncCallback if previously set&#x20;

NOTE: calling this on an IDR Access Unit is undefined behaviour

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the Access Unit to be skipped&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_DecodeSkip( DIL_Decoder              decoder,
                               uint16_t                 input_cc,
                               int64_t                  timestamp );
```

### DIL\_Flush

Synchronize client and DIL by throwing away any pending decodes and renders and waiting for all callbacks to complete

NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called with a the result filed in the callback will indicate the error DIL\_RC\_Flushed making DIL calls in this callback could result in a deadlock use the DIL\_SetFlushAsyncCallback to break this loop

* **decoder**             DIL instance
* **return**                 DIL API return code

```cpp
DIL_ReturnCode DIL_Flush( DIL_Decoder              decoder );
```

### DIL\_Drain

synchronize client and DIL by allowing any pending decodes and renders to complete and waiting for all callbacks to complete

NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called making DIL calls in this callback could result in a deadlock use the DIL\_SetDrainAsyncCallback to break this loop

* **decoder**             DIL instance
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Drain( DIL_Decoder              decoder );
```

### DIL\_GetGLContext

Get the DIL's GL Context

* **decoder** DIL instance&#x20;
* **context** pointer to pointer of GL Context
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetGLContext( DIL_Decoder            decoder,
                                 void**                 context );
```

### DIL\_GetGLDisplay

Get the DIL's GL Display

* **decoder** DIL instance&#x20;
* **display** pointer to pointer of GL Display
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetGLDisplay( DIL_Decoder            decoder,
                                 void**                 display);
```

### DIL\_SetGLWindow

Set the DIL's GL output window

* **decoder** DIL instance
* **gl\_external\_window** pointer to destination window
* **is\_secure** true if the destination window is secure
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_SetGLWindow( DIL_Decoder             decoder,
                                void*                   gl_external_window,
                                bool                    is_secure );
```

### DIL\_SetExitRequestedCallback

Set ExitRequest call back function

* **decoder** DIL instance&#x20;
* **callback** pointer to DILOnExitRequestedCallBack function&#x20;
* **user\_data** pointer to client's user data&#x20;
* **return** DIL API return code

```cpp
typedef void (*DILOnExitRequestedCallBack)( void* user_data );

DIL_ReturnCode DIL_SetExitRequestedCallback( DIL_Decoder                    decoder,
                                             DILOnExitRequestedCallBack     callback,
                                             void*                          user_data);

```

### DIL\_SetDecodedAsyncCallback

Set asynchronous on decoded call back function

* **result** result of the decode request, see DIL\_ReturnCode&#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was decoded&#x20;

* **input** pointer to base input picture, the same passed in the previous DIL\_Decode&#x20;

* **output** pointer to LCEVC enhanced (or base pass-through) output picture, the same passed in the previous DIL\_Decode&#x20;

* **information** pointer to decoder information structure (not the same passed in from the previous DIL\_Decode)&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetDecodedAsyncCallback

* **decoder** DIL instance&#x20;

* **callback** pointer to DILOnDecodedAsyncCallBack function&#x20;

* **user\_data** pointer to client's user data&#x20;

* **return** DIL API return code

```cpp
typedef void (*DILOnDecodedAsyncCallBack)( DIL_ReturnCode                   result,
                                           uint16_t                         input_cc,
                                           int64_t                          timestamp,
                                           DIL_Image                        input,
                                           DIL_Image                        output,
                                           const DIL_DecodeInformation*     information,
                                           void*                            user_data );

DIL_ReturnCode DIL_SetDecodedAsyncCallback( DIL_Decoder                     decoder,
                                            DILOnDecodedAsyncCallBack       callback,
                                            void*                           user_data );
```

### DIL\_SetRenderAsyncCallback

Set asynchronous on render callback function

* **result** result of render request, see DIL\_ReturnCode&#x20;

* **completion\_time\_us** actual time in microseconds the render completed &#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was rendered&#x20;

* **output** pointer to the same picture passed in with the DIL\_Render method&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetRenderAsyncCallback

* **decoder** DIL instance&#x20;

* **callback** pointer to DILOnRenderAsyncCallBack function&#x20;

* **user\_data** pointer to client's user data&#x20;

* **return** DIL API return code

```cpp
typedef void (*DILOnRenderAsyncCallBack)( DIL_ReturnCode          result,
                                          int64_t                 completion_time_us,
                                          uint16_t                input_cc,
                                          int64_t                 timestamp,
                                          DIL_Image               output,
                                          void*                   user_data );
                                          
DIL_ReturnCode DIL_SetRenderAsyncCallback( DIL_Decoder                      decoder,
                                           DILOnRenderAsyncCallBack         callback,
                                           void*                            user_data );
```

### DIL\_SetRenderResizeAsyncCallback

Called at the start of the render process when the image to be displayed has a different size to the last image rendered. This is a blocking call and there should be no DIL calls in the callback function.&#x20;

* **result** result of render request, see DIL\_ReturnCode&#x20;

* **completion\_time\_us** actual time in microseconds the render completed &#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was rendered&#x20;

* **output** pointer to the same picture passed in with the DIL\_Render method&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetRenderAsyncCallback

* **decoder** DIL instance

* **callback** pointer to DILOnRenderAsyncCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnRenderAsyncCallBack)( DIL_ReturnCode          result,
                                          int64_t                 completion_time_us,
                                          uint16_t                input_cc,
                                          int64_t                 timestamp,
                                          DIL_Image               output,
                                          void*                   user_data );

DIL_ReturnCode DIL_SetRenderResizeAsyncCallback( DIL_Decoder                      decoder,
                                                 DILOnRenderAsyncCallBack         callback,
                                                 void*                            user_data );

```

### DIL\_SetFlushAsyncCallback

Callback function for the DIL\_Flush() function

* **user\_data** pointer to client's user data

* **decoder** DIL instance

* **callback** pointer to DILOnSynchronizeCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnSynchronizeCallBack)( void* user_data );

DIL_ReturnCode DIL_SetFlushAsyncCallback( DIL_Decoder                      decoder,
                                          DILOnSynchronizeCallBack         callback,
                                          void*                            user_data );
```

### DIL\_SetDrainAsyncCallback

Callback function for the DIL\_Drain() function

* **user\_data** pointer to client's user data

* **decoder** DIL instance

* **callback** pointer to DILOnSynchronizeCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnSynchronizeCallBack)( void* user_data );

DIL_ReturnCode DIL_SetDrainAsyncCallback( DIL_Decoder                      decoder,
                                          DILOnSynchronizeCallBack         callback,
                                          void*                            user_data );
```

### DIL\_InputMouseButtonState

Pass the DIL the position of the input mouse button

* **decoder** DIL instance&#x20;
* **pressed** true if button is pressed&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_InputMouseButtonState( DIL_Decoder             decoder,
                                          bool                    pressed );
```

### DIL\_InputMousePosition

Pass the DIL the position of the input mouse&#x20;

* **decoder** DIL instance&#x20;
* **x** horizontal coordinate of the mouse cursor&#x20;
* **y** vertical coordinate of the mouse cursor&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_InputMousePosition( DIL_Decoder                decoder,
                                       float                      x,
                                       float                      y );
```

## DIL Types and Enumerations

### DIL\_NALFormat enum

DIL\_NALFormat represents the two possible NAL Unit formats, for each of the MPEG base codecs the DIL currently supports: H264 (ISO/IEC 14496-10, aka AVC) and H265 (ISO/IEC 23008-2, aka HEVC) NAL, where Annex B indicates 3 byte 0x000001 or 4 byte 0x00000001 start code prefix as described in Annex B of (H264 or HEVC), LP (Length Prefix) indicates a 4 byte prefix with the length of the NAL unit as MSBF 32 bit unsigned int.&#x20;

```cpp
typedef enum DIL_NALFormat
{
    DIL_UnknownNALFormat = 0,
    DIL_H264_AnnexB,
    DIL_H265_AnnexB,
    DIL_H264_LP,
    DIL_H265_LP
} DIL_NALFormat;
```

### DIL\_ReturnCode enum

DIL\_ReturnCode represents the possible API return codes. Most of the API calls will return one of these values.

```cpp
typedef enum DIL_ReturnCode
{
    DIL_RC_Success      =  0,       /* The API call completed successfully */
    DIL_RC_Error        = -1,       /* A generic catch all error */
    DIL_RC_InvalidParam = -2,       /* The user supplied an invalid parameter to the function call */
    DIL_RC_NotFound     = -3,       /* The query call failed to find the item by name, but didn't not generate an error. */
    DIL_RC_NotSupported = -4,       /* The functionality requested is not supported on the running system. */
    DIL_RC_Flushed      = -5,       /* The requested operation failed because it was flushed via DIL_Flush. */
    DIL_RC_Timeout      = -6        /* The requested operation failed because it timedout. */
} DIL_ReturnCode;
```

### DIL\_ColorFormat enum

This enum represents the supported colour formats, currently only 8 bit per sample formats are supported For a detailed description of the formats see: <https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html>

```cpp
typedef enum DIL_ColorFormat
{
    DIL_UnknownColorFormat = 0,     /* Unspecified colour format */
    DIL_I420_8,                     /* 8 bit 4:2:0 YUV Planar colour format: Y, U and V on separate planes */
    DIL_NV12_8,                     /* 8 bit 4:2:0 YUV SemiPlanar colour format: Y plane and UV interleaved plane */
    DIL_RGB_8,                      /* 8 bit Interleaved R, G, B planes 24 bit per sample */
    DIL_RGBA_8,                     /* 8 bit Interleaved R, G, B and A planes 32 bit per sample */
    DIL_I420_10LE,                  /* 10 bit Little Endian 4:2:0 YUV Planar colour format: Y, U and V on separate planes */

} DIL_ColorFormat;

```

### DIL\_ImageType enum

This enum identifies the type of planes the DIL\_Image consists of; planes are non-overlapping sets of samples belonging to one or more colour channels with a constant sample stride, so RGB is a single plane, because R G and B pointed at separately would overlap UV from NV12 is a single plane, because U and V pointed at separately would overlap Y U and V from I420 are three planes because they do not overlap

```cpp
typedef enum DIL_ImageType
{
    DIL_UnknownImageType = 0,       /* Unspecified image type */
    DIL_Buffer_Array,               /* One buffer per plane, even if image is on a single memory allocation, an array of planes shall be set by assigning the right pointers to plane buffers. DIL allocated buffers have contiguous memory allocation */
    DIL_Texture_Array,              /* One texture per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
    DIL_HardwareBuffer_Array        /* One HardwareBuffer per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
} DIL_ImageType;

```

### DIL\_ImageDesc struct

Holds configurations of a single DIL\_Image instance

```cpp
typedef struct DIL_ImageDesc
{
    bool            is_managed;     /* True means that Image (and its memory) is managed by DIL  */
    DIL_ImageType   image_type;     /* Whether it is a buffer array or a texture array */
    DIL_ColorFormat color_format;   /* Colour format */
	  uint32_t        width;          /* Nominal net width of the image in luma samples, i.e. no alignment to macroblocks or striding */
    uint32_t        height;         /* Nominal net height of the image in luma samples, i.e. no sliceheight or padding */
    bool            can_modify;     /* Specifying wether content of image is readOnly or changable */
    bool            can_resize;     /* Specifying wether allocated memory is resizable or not. (Does not apply to DIL_Buffer_External) */
    uint32_t        options;        /* options used in creating the Image, in the case of HardwareBuffers this is a combination of  DIL_HardwareBufferUsageOption */
} DIL_ImageDesc;
```

### DIL\_ContextSettings struct

This structure holds parameters related to the window surface the DIL would render into, in case of on screen rendering those parameters are normally provided by the windowing system, e.g. EGL, GLFW.

```cpp
typedef struct DIL_ContextSettings
{
    void* gl_external_window;       /* Surface window, from the windowing system */
    void* gl_external_context;      /* Context, from the windowing system */
    bool  is_secure;                /* True if the client has initialised window and context as secure for protected content */
} DIL_ContextSettings;

```

### DIL\_RenderInformation struct

This structure captures the properties related to the (on screen) window rendering, they do not apply for buffer or texture output (off screen). NOTE: rotation is a per frame property because it may change frame by frame due to for ex. the encoding device rotating while shooting.

```cpp
typedef struct DIL_RenderInformation
{
    uint32_t rotation;                  /* Degrees of rotation for the final rendering on the destination window, in 90 unit increments clockwise */
    float    pixel_aspect_ratio_num;    /* Pixel Aspect Ratio of width to height, this is the numerator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
    float    pixel_aspect_ratio_den;    /* Pixel Aspect Ratio of width to height, this is the denominator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
} DIL_RenderInformation;
```

### DIL\_DecodeInformation struct

This structure captures properties related to the decoding process, width and height of the output, whether LCEVC data has been found and whether it has been applied.

```cpp
typedef struct DIL_DecodeInformation
{
    uint32_t width;                     /* width of the decoded frame */
    uint32_t height;                    /* height of the decoded frame */
    bool     skip_requested;            /* decode skip was requested */
    bool     lcevc_enhanced;            /* true if the frame has been enhanced by lcevc decoding */
    bool     lcevc_available;           /* true if lcevc encoded data is available for this frame */
} DIL_DecodeInformation;
```

### DIL\_ImagePlaneBufferDesc struct

This struct describes a plane memory buffer

```cpp
typedef struct DIL_ImagePlaneBufferDesc
{
    uint8_t* data;                      /* Pointer to first byte of the first active sample of the plane buffer, NOTE: active means no cropping, the active origin is 0,0 */
    uint32_t sample_byte_stride;        /* Distance in bytes between consecutive sample values of the same channel, for ex. UV plane on 8bit depth this will be 2 */
    uint32_t row_byte_stride;           /* Distance in bytes between the first sample of two consecutive rows */
} DIL_ImagePlaneBufferDesc;

```

### DIL\_ImagePlaneTextureDesc struct

This struct describes a plane OpenGL texture

```cpp
typedef struct DIL_ImagePlaneTextureDesc
{
    uint32_t target;                /* OpenGL/GLES texture target, as in glBindTexture target */
    uint32_t id;                    /* OpenGL/GLES texture id. Unused indexes should be set to 0 */
    uint32_t width;                 /* OpenGL/GLES texture width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* OpenGL/GLES texture height, which might have extra padding, NOTE: the active height is in the parent Image struct */
    int32_t  internal_format;       /* OpenGL/GLES (GLint) sized internal format as from Table 2 of glTexImage2D man page */
} DIL_ImagePlaneTextureDesc;

```

### DIL\_HardwareBufferUsageOption enum

This enum use used to set the UsageOption when creating HardwareBuffer DIL\_Image types. It should be provided in the "hardware\_buffer\_usage\_option" tag when configuring the DIL object

```cpp
typedef enum DIL_HardwareBufferUsageOption
{
    UsageOption_None = 0,

    UsageOption_Read_Never  = (1 << 0),
    UsageOption_Read_Rarely = (1 << 1),
    UsageOption_Read_Often  = (1 << 2),

    UsageOption_Write_Never  = (1 << 4),
    UsageOption_Write_Rarely = (1 << 5),
    UsageOption_Write_Often  = (1 << 6),

    UsageOption_Protected_Content = (1 << 7),

    UsageOption_GPU_Read          = (1 << 8),
    UsageOption_GPU_Write         = (1 << 9)
} DIL_HardwareBufferUsageOption;

```

### DIL\_ImagePlaneHardwareBufferDesc struct

This struct describes a plane HardwareBuffer

```cpp
typedef struct DIL_ImagePlaneHardwareBufferDesc
{
    uint32_t width;                 /* width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* height, which might have extra padding, NOTE: the active height is in the parent Image struct */
		void*    nativeObject;          /* platform native object/id for the HardwareBuffer */
} DIL_ImagePlaneHardwareBufferDesc;

```

## DIL Properties API

### DIL\_QueryProperty

Obtain a specific named property. This function provides a way of looking up a property by name to observe it's current value, this may be of use for a user to query DIL properties

```cpp
DIL_ReturnCode DIL_QueryProperty(DIL_Decoder decoder, const char* name, DILProperty* output);
```

### DIL\_SetProperty

Set a specific named property. This function provides a way of changing a property by name

```cpp
DIL_ReturnCode DIL_SetProperty(DIL_Decoder decoder, const char* name, DILProperty* value);
```

### DIL\_QueryPropertyGroup

Obtain all the available properties for a named group

```cpp
DIL_ReturnCode DIL_QueryPropertyGroup(DIL_Decoder decoder, const char* name, DILPropertyGroup* output);
```

### DIL\_ReleaseProperty

Release a previously queried property, this function provides a way of releasing any memory allocated by a call to DIL\_QueryProperty. Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleaseProperty(DIL_Decoder decoder, DILProperty* value);
```

### DIL\_ReleasePropertyGroup

Release a previously queried property group, this function provides a way of releasing any memory allocated by a call to DIL\_QueryPropertyGroup Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleasePropertyGroup(DIL_Decoder decoder, DILPropertyGroup* value);
```

### DILProperty struct.

This structure contains all relevant information used to query a property or metadata from the DIL.

```cpp
typedef struct DILProperty
{
  const char*     name;        /* The name of the property.  */
  const char*     description; /* A description of the property. */
  DILPropertyType type;        /* The value type for this property. */
  union
  {
    int8_t      i8;
    int16_t     i16;
    int32_t     i32;
    int64_t     i64;
    uint8_t     u8;
    uint16_t    u16;
    uint32_t    u32;
    uint64_t    u64;
    float       f;
    double      d;
    const char* str;
    const void* vptr;
    const void* data;
		const DILPropertyGroup* pgptr;
  } value;                  /* A union representing the value of the property. The user must read the field that match the type enum value. */

  const uint8_t* blob;      /* Special field containing a block of contiguous memory when the type enum is set to DIL_PT_Blob. This is allowed to be NULL. */
  uint32_t       blob_size; /* When the blob is not NULL then the length in bytes of the memory, otherwise 0. */
	uint64_t       data_size; /* size of the "data" buffer in bytes */
	bool           can_free;  /* flag to show if the pointer values should be free'ed when the DIL_ReleaseProperty is called */
} DILProperty;
```

### DILPropertyGroup struct.

This structure contains a logical grouping of properties, this is a convenience feature to assist with providing a "pretty" command line, or GUI.

All properties are assigned to a group, and properties can only appear in one group.

```cpp
typedef struct DILPropertyGroup_t
{
  const char*  name;           /* The name for this group of properties. */
  const char*  description;    /* A description of the property group. */
  DILProperty* properties;     /* An array of properties for this group. */
  uint32_t     property_count; /* The length of the properties array. */
	bool         can_free;       /* flag to show if the properties pointer should be free'ed when the DIL_ReleaseProperty is called */
} DILPropertyGroup;
```


# API Overview

Using the DIL is fairly easy, especially if compared to the more basic DPI, since it operates at a relatively high abstraction level. Apart from obvious creation and destruction of a DIL instance (respectively `DIL_Create()` and `DIL_Destroy()`), the main API calls are effectively two:

* Feed input - `DIL_AddNALData()`
* Decode - `DIL_DecodeandRender()` (for decoding on screen, alternatively `DIL_Decode()` for decoding off screen followed by `DIL_Render()`)
* Feed input - `DIL_AddNALData()`
* Decode - `DIL_DecodeandRender()` (for decoding on screen, alternatively `DIL_Decode()` for decoding off screen followed by `DIL_Render()`)

The DIL also offers an API for the client to retrieve the size of the output picture, which can be useful to pre-allocate buffers or textures of the right size before decoding:

* Get size information - `DIL_GetDecodeInformation()`

Note: input buffers must have NAL byte stream format with either Annex B or Length Prefix format.

### Drop/Skip frames

LCEVC content may be encoded with a temporal feature that requires each and every frame to be actioned on in order to keep the DPI internal temporal reference data correct. As a result when the client skips frames, for ex. because of a seek in the timeline, or drops frames because they are “late”, it shall let the DIL know by calling the following function:

* `DIL_DecodeSkip()`

The DIL does know whether the temporal feature is on and therefore can fall back to a no operation case if that is the case. It is not advised that the client tries to implement this behaviour on its side.

### Synchronous and Asynchronous modes

The DIL can be used in both synchronous or asynchronous mode. This is applicable to the decode and render functions:

* `DIL_Decode()`
* `DIL_DecodeAndRender()`
* `DIL_DecodeSkip()`
* `DIL_Render()`

If the client has previously set the callback function by means of the `DIL_SetDecodedAsyncCallback()` or `DIL_SetRenderAsyncCallback()` API calls, the above decode functions will work asynchronously, therefore returning immediately and triggering the client’s callback function when the decode (or render) has completed. If the callback has not been set the functions will work synchronously.

### Base pass-through mode

The DIL also supports a pass-through mode, in which the base is simply copied on to the output without applying the LCEVC enhancement. This mode is triggered when no LCEVC data is found for the frame and can be set as forced behaviour, for every frame, from the JSON configuration.


# Example Integration Code

In basic terms, a sample integration code using the DIL can be briefly described, among the various phases, as follows:

```cpp
// Include DIL header
#include <lcevc_dil.h>
// Initialisation phase ///////////////////////////////////////////////////////
// Create an instance of the DIL
DIL_ContextSettings context_settings = { 0 };
char * json_settings = "{\
 \"gl_decode\": true,\
 \"gl_es\": true,\
 \"gl_major\": 3,\
 \"gl_minor\": 1,\
 \"swap_interval\": 0,\
 \"fullscreen\": false,\
 \"hardware_buffers\": false,\
 \"use_u8_surface\": false,\
 \"force_passthrough\": false\
}";
// Create a DIL instance
DIL_Decoder decoder = nullptr;
if (DIL_Create(json_settings, &context_settings , &decoder) != DIL_RC_Success)
{
    fprintf(stderr, "Unable to create DIL instance\n");
    exit(-1);
}
// Destruction phase //////////////////////////////////////////////////////////
DIL_Destroy(decoder);
decoder = nullptr;
// Handling input buffers /////////////////////////////////////////////////////
//...
DIL_NALFormat nalFormat;
// Set NAL format, from metadata, properties or parsing the input buffers
// ...
if (DIL_AddNALData(decoder, inputCc, presentationTimeStamp, buffer, length, nalFormat) != DIL_RC_Success)
{
    fprintf(stderr, "Unable to pass input buffer for inputCc=%u pts=%PRId64\n", inputCc, presentationTimeStamp);
    exit(-2);
}
// Handling decoded frame from base decoder ///////////////////////////////////
// ...
// Get size of ouput picture if needed
DIL_DecodeInformation decodeInfo;
if (DIL_GetDecodeInformation(decoder, inputCc, presentationTimeStamp, baseWidth, baseHeight, &decodeInfo) != DIL_RC_Success)
{
    fprintf(stderr, "DIL get decode info failed for inputCc=%u pts=%PRId64\n", inputCc, baseImage.pts;
    exit(-3);
}
// Using decode info
// ...
DIL_Image baseImage;
// Fill the base "Image" struct with the data properties of the base frame
// ...
DIL_RenderInformation renderInfo;
// Fill rotation and pixel aspect ratio
// ...
if (DIL_DecodeAndRender(decoder, inputCc, presentationTimeStamp, baseImage, &renderInfo, &decodeInfo) != DIL_RC_Success)
{
    fprintf(stderr, "DIL decode failed for pts=%PRId64\n", baseImage.pts;
    exit(-4);
}
// Decode info reports if LCEVC is available and has been applied
// ...
```


# DIL API

### DIL\_Create

Create an instance of a DIL Decoder

* **json\_settings** JSON string with initialisation parameters
* **context\_settings** settings from the windowing system&#x20;
* **instance decoder** instance created
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Create( const char*                   json_settings,
                           const DIL_ContextSettings*    context_settings,
                           DIL_Decoder                   instance );
```

### DIL\_Destroy

Destroy an instance of a DIL Decoder

* **decoder** instance to be destroyed&#x20;

```cpp
void DIL_Destroy( DIL_Decoder decoder);
```

### DIL\_GetImage

Returns a fresh usable instance of DIL\_Image

* **decoder** DIL instance
* **image\_desc** Describing Image configurations
* **image** Result DIL\_Image if creation is successful, NULL otherwise
* **return** DIL\_RC\_Error if image\_desc contains incompatible values &#x20;

```cpp
DIL_ReturnCode DIL_GetImage(DIL_Decoder decoder, DIL_ImageDesc* image_desc, DIL_Image* image);
```

### **DIL\_ReleaseImage**

Releases instance of DIL\_Image&#x20;

NOTE: No reference calls to image address should be made after this method

* **decoder** DIL instance&#x20;
* **image** DIL\_Image instance to be released&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ReleaseImage(DIL_Decoder decoder, DIL_Image image);
```

### DIL\_ImageGetDesc

Will return DIL\_ImageDesc of the image

* **image** DIL\_Image to query&#x20;
* **desc** Contents of the pointer will be rewritten with image descriptions&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ImageGetDesc(DIL_Image image, DIL_ImageDesc* desc);
```

### DIL\_ImageSetActiveRegion

Will set the active region for an image

* **image**           DIL\_Image instance to modify
* **offset\_x**        X offset for the region
* **offset\_y**        Y offset for the region
* **width**             width for the region
* **height**            height for the region
* **return**            DIL\_RC\_Error if crop region is out of bounds, image\_type doesn't support cropping

```cpp
DIL_ReturnCode DIL_ImageSetActiveRegion( DIL_Image                  image,
                                         uint32_t                   offset_x,
                                         uint32_t                   offset_y,
                                         uint32_t                   width,
                                         uint32_t                   height);
```

### DIL\_ImageGetActiveRegion

Will get the active region for an image

* **image**           DIL\_Image instance to modify
* **offset\_x**        X offset for the region
* **offset\_y**        Y offset for the region
* **width**            width for the region
* **height**           height for the region
* **return**            DIL\_RC\_Error if crop region is out of bounds, image\_type doesn't support cropping

```cpp
DIL_ReturnCode DIL_ImageGetActiveRegion( DIL_Image                  image,
                                         uint32_t*                  offset_x,
                                         uint32_t*                  offset_y,
                                         uint32_t*                  width,
                                         uint32_t*                  height);
```

### DIL\_ImageGetPlanesCount

Will return number image's number of planes

* **image** DIL\_Image to query&#x20;
* **plane\_count** Will be filled with number of planes&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_ImageGetPlanesCount(DIL_Image image, uint32_t* plane_count);
```

### DIL\_ImageSetPlaneBuffer

Will set content and configurations of a plane by data buffer

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Buffer or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneBuffer( DIL_Image	                  image,
                                        uint32_t                    plane_index,
                                        DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImageGetPlaneBuffer

Will return content and configurations of a plane

* **image** DIL\_Image to query&#x20;
* **plane\_index** Index of the plane to query&#x20;
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to write to&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_Buffer

```cpp
DIL_ReturnCode DIL_ImageGetPlaneBuffer( DIL_Image                   image,
                                        uint32_t                    plane_index,
                                        DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImageSetPlaneTexture

Will set content and configurations of a plane by OpenGL/GLES texture

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **texture\_desc** DIL\_ImagePlaneTextureDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneTexture( DIL_Image                  image,
                                         uint32_t                   plane_index,
                                         DIL_ImagePlaneTextureDesc* texture_desc);
```

### DIL\_ImageGetPlaneTexture

Will return content and configurations of a plane by OpenGL/GLES texture

* **image** DIL\_Image instance to modify&#x20;
* **plane\_index** Index of the plane to modify&#x20;
* **texture\_desc** DIL\_ImagePlaneTextureDesc instance to read from&#x20;
* **return** DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageGetPlaneTexture( DIL_Image                  image,
                                         uint32_t                   plane_index,
                                         DIL_ImagePlaneTextureDesc* texture_desc);
```

### DIL\_ImageSetPlaneHardwareBuffer

Will set content and configurations of a plane by OS specific Hardware buffer.

* **image**               DIL\_Image instance to modify
* **plane\_index**    Index of the plane to modify
* **desc**                  DIL\_ImagePlaneHardwareBufferDesc instance to read from
* **return**                DIL\_RC\_Error if plane\_index is out of bounds, image\_type is not DIL\_Texture or image is managed

```cpp
DIL_ReturnCode DIL_ImageSetPlaneHardwareBuffer( DIL_Image                         image,
                                                uint32_t                          plane_index,
                                                DIL_ImagePlaneHardwareBufferDesc* desc);
```

### DIL\_ImageGetPlaneHardwareBuffer

Will return content and configurations of a plane by OS specific Hardware buffer.

* **image** DIL\_Image instance to query
* **plane\_index** Index of the plane to query
* **desc** DIL\_ImagePlaneHardwareBufferDesc instance to write to
* **return** Will return DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImageGetPlaneHardwareBuffer( DIL_Image                         image,
                                                uint32_t                          plane_index,
                                                DIL_ImagePlaneHardwareBufferDesc* desc);
```

### DIL\_ImageSetUserData

Will set the user data associated with the image

* **image**  DIL\_Image to set the userdata for
* **userdata**   The userdata to associate with the image
* **return**         DIL\_RC\_InvalidParam if the image, DIL\_RC\_Success otherwise

```cpp
DIL_ReturnCode DIL_ImageSetUserData(DIL_Image image, void* userdata);
```

### DIL\_ImageGetUserData

Will get the user data associated with the image through DIL\_ImageSetUserData

* **image**         DIL\_Image to get the userdata for
* **userdata**    The userdata to associate with the image
* **return**         DIL\_RC\_InvalidParam if the image is invalid or userdata is invalid, DIL\_RC\_Success otherwise

```cpp
DIL_ReturnCode DIL_ImageGetUserData(DIL_Image image, void** userdata);
```

### DIL\_ImagePlaneHardwareBufferLock

Will return content and configurations of a plane buffer obtained by locking the hardware buffer for CPU access

* **image** DIL\_Image to access
* **plane\_index** Index of the plane to query
* **read\_access** Whether we want read access
* **write\_access** Whether we want write access
* **buffer\_desc** DIL\_ImagePlaneBufferDesc instance to write to
* **return** Will return DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImagePlaneHardwareBufferLock( DIL_Decoder                 decoder,
                                                 DIL_Image                   image,
                                                 uint32_t                    plane_index,
                                                 bool                        read_access,
                                                 bool                        write_access,
                                                 DIL_ImagePlaneBufferDesc*   buffer_desc);
```

### DIL\_ImagePlaneHardwareBufferUnlock

Will unlock the hardware buffer plane once CPU access is completed.

* **image** DIL\_Image to unlock
* **plane\_index** Index of the plane to query
* **return** Returns DIL\_RC\_Error if plane\_index is out of bounds or image\_type is not DIL\_HardwareBuffer

```cpp
DIL_ReturnCode DIL_ImagePlaneHardwareBufferUnlock( DIL_Decoder                 decoder,
                                                   DIL_Image                   image,
                                                   uint32_t                    plane_index);
```

### DIL\_AddTSData

Feed Transport Stream packet(s) for the DIL to extract and store LCEVC data, feed a buffer of TS packets associated with the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **data** pointer to the TS buffer&#x20;
* **size** size of the TS buffer&#x20;
* **returns** DIL API return code

```cpp
DIL_ReturnCode DIL_AddTSData( DIL_Decoder           decoder,
                              uint16_t              input_cc,
                              uint8_t*              data,
                              uint32_t              size );
```

### DIL\_AddWebmData

Feed Webm payload for the DIL to extract and store LCEVC data

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **data** pointer to the TS buffer&#x20;
* **size** size of the TS buffer&#x20;
* **returns** DIL API return code

```cpp
DIL_ReturnCode DIL_AddWebmData( DIL_Decoder         decoder,
                                uint16_t            input_cc,
                                uint8_t*            data,
                                uint32_t            size );
```

### DIL\_AddNALData

Feed NAL unit(s) for the DIL to extract and store LCEVC data, feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed NAL units&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **nalu\_format** format of the NAL units&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_AddNALData( DIL_Decoder          decoder,
                               uint16_t             input_cc,
                               int64_t              timestamp,
                               uint8_t*             data,
                               uint32_t             size,
                               DIL_NALFormat        nalu_format );
```

### DIL\_AddNALDataEx

Feed NAL unit(s) for the DIL to extract and store LCEVC data (Extended), feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp and additionally let the DIL strip the LCEVC payloads and zero pad the trailing bytes, returning the new size before padding&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed NAL units&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **nalu\_format** format of the NAL units
* **return** size of the stripped NAL units buffer

```cpp
uint32_t DIL_AddNALDataEx( DIL_Decoder              decoder,
                           uint16_t                 input_cc,
                           int64_t                  timestamp,
                           uint8_t*                 data,
                           uint32_t                 size,
                           DIL_NALFormat            nalu_format );
```

### DIL\_AddRawData

Feed LCEVC data to the DIL, feed a buffer of pre parsed LCEVC payload data for the Access Unit identified by input cc and timestamp&#x20;

NOTE: input\_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback&#x20;

NOTE: input\_cc shall have monotonically increasing value over calls&#x20;

NOTE: timestamp shall have monotonically increasing value over calls within the same input\_cc

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the passed LCEVC data&#x20;
* **data** pointer to the NAL units buffer&#x20;
* **size** size of the NAL units buffer&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_AddRawData( DIL_Decoder          decoder,
                               uint16_t             input_cc,
                               int64_t              timestamp,
                               uint8_t*             data,
                               uint32_t             size );
```

### DIL\_GetDecodeInformation

Get details about the decoding process for the Access Unit identified by input cc and timestamp

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture&#x20;
* **base\_width** width of the base decoded picture&#x20;
* **base\_height** height of the base decoded picture&#x20;
* **information** pointer to decoder information structure that the DIL will fill&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetDecodeInformation( DIL_Decoder                    decoder,
                                         uint16_t                       input_cc,
                                         int64_t                        timestamp,
                                         uint32_t                       base_width,
                                         uint32_t                       base_height,
                                         DIL_DecodeInformation*         information );
```

### DIL\_Decode

Decode on to an off screen DIL\_Image, get an off screen LCEVC enhanced (or base pass-through) picture from the input base picture relative to input cc and timestamp

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be decoded&#x20;
* **max\_decode\_time\_us maximum** amount of time the decode has to complete, use 0 to disable decode timeout
* **input** pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)&#x20;
* **output** pointer to LCEVC enhanced (or base pass-through) output picture&#x20;
* **decode\_information** pointer to decoder information structure that the DIL will fill (if DIL\_SetDecodedAsyncCallback has not been called)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Decode( DIL_Decoder                      decoder,
                           uint16_t                         input_cc,
                           int64_t                          timestamp,
                           uint32_t                         max_decode_time_us,
                           DIL_Image*                       input,
                           DIL_Image*                       output,
                           DIL_DecodeInformation*           decode_information );
```

### DIL\_Render

Render to back buffer a previously decoded DIL\_Image

NOTE: for the scheduled render to be set DIL\_SetRenderAsyncCallback must have been previously called

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be rendered&#x20;
* **image** pointer to a picture to be rendered to back buffer&#x20;
* **render\_information** pointer to a render information structure&#x20;
* **delay\_us** delay in microseconds for the render to eb executed (0 for no delay)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Render( DIL_Decoder                      decoder,
                           uint16_t                         input_cc,
                           int64_t                          timestamp,
                           DIL_Image                        image,
                           const DIL_RenderInformation*     render_information,
                           int64_t                          delay_us );
```

### DIL\_DecodeAndRender

Decode and render immediately to back buffer

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference of the picture to be decoded and rendered&#x20;
* **max\_decode\_time\_us maximum** amount of time the decode has to complete, use 0 to disable decode timeout
* **input** pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)&#x20;
* **render\_information** pointer to a render information structure&#x20;
* **decode\_information** pointer to decoder information structure that the DIL will fill (if DIL\_SetDecodedAsyncCallback has not been called)&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_DecodeAndRender( DIL_Decoder                         decoder,
                                    uint16_t                            input_cc,
                                    int64_t                             timestamp,
                                    uint32_t                            max_decode_time_us,
                                    DIL_Image                           input,
                                    const DIL_RenderInformation*        render_information,
                                    DIL_DecodeInformation*              decode_information );
```

### DIL\_DecodeSkip

Let the DIL know that an Access Unit is not being presented (dropped or skipped), DIL will do the minimum processing to keep internal state consistent for the next picture&#x20;

NOTE: this call will generate a DecodeAsyncCallback if previously set&#x20;

NOTE: calling this on an IDR Access Unit is undefined behaviour

* **decoder** DIL instance&#x20;
* **input\_cc** input continuity counter&#x20;
* **timestamp** time reference for the Access Unit to be skipped&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_DecodeSkip( DIL_Decoder              decoder,
                               uint16_t                 input_cc,
                               int64_t                  timestamp );
```

### DIL\_Flush

Synchronize client and DIL by throwing away any pending decodes and renders and waiting for all callbacks to complete

NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called with a the result filed in the callback will indicate the error DIL\_RC\_Flushed making DIL calls in this callback could result in a deadlock use the DIL\_SetFlushAsyncCallback to break this loop

* **decoder**             DIL instance
* **return**                 DIL API return code

```cpp
DIL_ReturnCode DIL_Flush( DIL_Decoder              decoder );
```

### DIL\_Drain

synchronize client and DIL by allowing any pending decodes and renders to complete and waiting for all callbacks to complete

NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called making DIL calls in this callback could result in a deadlock use the DIL\_SetDrainAsyncCallback to break this loop

* **decoder**             DIL instance
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_Drain( DIL_Decoder              decoder );
```

### DIL\_GetGLContext

Get the DIL's GL Context

* **decoder** DIL instance&#x20;
* **context** pointer to pointer of GL Context
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetGLContext( DIL_Decoder            decoder,
                                 void**                 context );
```

### DIL\_GetGLDisplay

Get the DIL's GL Display

* **decoder** DIL instance&#x20;
* **display** pointer to pointer of GL Display
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_GetGLDisplay( DIL_Decoder            decoder,
                                 void**                 display);
```

### DIL\_SetGLWindow

Set the DIL's GL output window

* **decoder** DIL instance
* **gl\_external\_window** pointer to destination window
* **is\_secure** true if the destination window is secure
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_SetGLWindow( DIL_Decoder             decoder,
                                void*                   gl_external_window,
                                bool                    is_secure );
```

### DIL\_SetExitRequestedCallback

Set ExitRequest call back function

* **decoder** DIL instance&#x20;
* **callback** pointer to DILOnExitRequestedCallBack function&#x20;
* **user\_data** pointer to client's user data&#x20;
* **return** DIL API return code

```cpp
typedef void (*DILOnExitRequestedCallBack)( void* user_data );

DIL_ReturnCode DIL_SetExitRequestedCallback( DIL_Decoder                    decoder,
                                             DILOnExitRequestedCallBack     callback,
                                             void*                          user_data);

```

### DIL\_SetDecodedAsyncCallback

Set asynchronous on decoded call back function

* **result** result of the decode request, see DIL\_ReturnCode&#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was decoded&#x20;

* **input** pointer to base input picture, the same passed in the previous DIL\_Decode&#x20;

* **output** pointer to LCEVC enhanced (or base pass-through) output picture, the same passed in the previous DIL\_Decode&#x20;

* **information** pointer to decoder information structure (not the same passed in from the previous DIL\_Decode)&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetDecodedAsyncCallback

* **decoder** DIL instance&#x20;

* **callback** pointer to DILOnDecodedAsyncCallBack function&#x20;

* **user\_data** pointer to client's user data&#x20;

* **return** DIL API return code

```cpp
typedef void (*DILOnDecodedAsyncCallBack)( DIL_ReturnCode                   result,
                                           uint16_t                         input_cc,
                                           int64_t                          timestamp,
                                           DIL_Image                        input,
                                           DIL_Image                        output,
                                           const DIL_DecodeInformation*     information,
                                           void*                            user_data );

DIL_ReturnCode DIL_SetDecodedAsyncCallback( DIL_Decoder                     decoder,
                                            DILOnDecodedAsyncCallBack       callback,
                                            void*                           user_data );
```

### DIL\_SetRenderAsyncCallback

Set asynchronous on render callback function

* **result** result of render request, see DIL\_ReturnCode&#x20;

* **completion\_time\_us** actual time in microseconds the render completed &#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was rendered&#x20;

* **output** pointer to the same picture passed in with the DIL\_Render method&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetRenderAsyncCallback

* **decoder** DIL instance&#x20;

* **callback** pointer to DILOnRenderAsyncCallBack function&#x20;

* **user\_data** pointer to client's user data&#x20;

* **return** DIL API return code

```cpp
typedef void (*DILOnRenderAsyncCallBack)( DIL_ReturnCode          result,
                                          int64_t                 completion_time_us,
                                          uint16_t                input_cc,
                                          int64_t                 timestamp,
                                          DIL_Image               output,
                                          void*                   user_data );
                                          
DIL_ReturnCode DIL_SetRenderAsyncCallback( DIL_Decoder                      decoder,
                                           DILOnRenderAsyncCallBack         callback,
                                           void*                            user_data );
```

### DIL\_SetRenderResizeAsyncCallback

Called at the start of the render process when the image to be displayed has a different size to the last image rendered. This is a blocking call and there should be no DIL calls in the callback function.&#x20;

* **result** result of render request, see DIL\_ReturnCode&#x20;

* **completion\_time\_us** actual time in microseconds the render completed &#x20;

* **input\_cc** input continuity counter&#x20;

* **timestamp** time reference of the picture that was rendered&#x20;

* **output** pointer to the same picture passed in with the DIL\_Render method&#x20;

* **user\_data** pointer to the same client's user data that was passed into the DIL\_SetRenderAsyncCallback

* **decoder** DIL instance

* **callback** pointer to DILOnRenderAsyncCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnRenderAsyncCallBack)( DIL_ReturnCode          result,
                                          int64_t                 completion_time_us,
                                          uint16_t                input_cc,
                                          int64_t                 timestamp,
                                          DIL_Image               output,
                                          void*                   user_data );

DIL_ReturnCode DIL_SetRenderResizeAsyncCallback( DIL_Decoder                      decoder,
                                                 DILOnRenderAsyncCallBack         callback,
                                                 void*                            user_data );

```

### DIL\_SetFlushAsyncCallback

Callback function for the DIL\_Flush() function

* **user\_data** pointer to client's user data

* **decoder** DIL instance

* **callback** pointer to DILOnSynchronizeCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnSynchronizeCallBack)( void* user_data );

DIL_ReturnCode DIL_SetFlushAsyncCallback( DIL_Decoder                      decoder,
                                          DILOnSynchronizeCallBack         callback,
                                          void*                            user_data );
```

### DIL\_SetDrainAsyncCallback

Callback function for the DIL\_Drain() function

* **user\_data** pointer to client's user data

* **decoder** DIL instance

* **callback** pointer to DILOnSynchronizeCallBack function

* **user\_data** pointer to client's user data

* **return** DIL API return code

```cpp
typedef void (*DILOnSynchronizeCallBack)( void* user_data );

DIL_ReturnCode DIL_SetDrainAsyncCallback( DIL_Decoder                      decoder,
                                          DILOnSynchronizeCallBack         callback,
                                          void*                            user_data );
```

### DIL\_InputMouseButtonState

Pass the DIL the position of the input mouse button

* **decoder** DIL instance&#x20;
* **pressed** true if button is pressed&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_InputMouseButtonState( DIL_Decoder             decoder,
                                          bool                    pressed );
```

### DIL\_InputMousePosition

Pass the DIL the position of the input mouse&#x20;

* **decoder** DIL instance&#x20;
* **x** horizontal coordinate of the mouse cursor&#x20;
* **y** vertical coordinate of the mouse cursor&#x20;
* **return** DIL API return code

```cpp
DIL_ReturnCode DIL_InputMousePosition( DIL_Decoder                decoder,
                                       float                      x,
                                       float                      y );
```


# DIL Types and Enumerations


# DIL Properties API

### DIL\_QueryProperty

Obtain a specific named property. This function provides a way of looking up a property by name to observe it's current value, this may be of use for a user to query DIL properties

```cpp
DIL_ReturnCode DIL_QueryProperty(DIL_Decoder decoder, const char* name, DILProperty* output);
```

### DIL\_SetProperty

Set a specific named property. This function provides a way of changing a property by name

```cpp
DIL_ReturnCode DIL_SetProperty(DIL_Decoder decoder, const char* name, DILProperty* value);
```

### DIL\_QueryPropertyGroup

Obtain all the available properties for a named group

```cpp
DIL_ReturnCode DIL_QueryPropertyGroup(DIL_Decoder decoder, const char* name, DILPropertyGroup* output);
```

### DIL\_ReleaseProperty

Release a previously queried property, this function provides a way of releasing any memory allocated by a call to DIL\_QueryProperty. Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleaseProperty(DIL_Decoder decoder, DILProperty* value);
```

### DIL\_ReleasePropertyGroup

Release a previously queried property group, this function provides a way of releasing any memory allocated by a call to DIL\_QueryPropertyGroup Any data contained in value maybe invalidated after this call

```cpp
DIL_ReturnCode DIL_ReleasePropertyGroup(DIL_Decoder decoder, DILPropertyGroup* value);
```

### DILProperty struct.

This structure contains all relevant information used to query a property or metadata from the DIL.

```cpp
typedef struct DILProperty
{
  const char*     name;        /* The name of the property.  */
  const char*     description; /* A description of the property. */
  DILPropertyType type;        /* The value type for this property. */
  union
  {
    int8_t      i8;
    int16_t     i16;
    int32_t     i32;
    int64_t     i64;
    uint8_t     u8;
    uint16_t    u16;
    uint32_t    u32;
    uint64_t    u64;
    float       f;
    double      d;
    const char* str;
    const void* vptr;
    const void* data;
		const DILPropertyGroup* pgptr;
  } value;                  /* A union representing the value of the property. The user must read the field that match the type enum value. */

  const uint8_t* blob;      /* Special field containing a block of contiguous memory when the type enum is set to DIL_PT_Blob. This is allowed to be NULL. */
  uint32_t       blob_size; /* When the blob is not NULL then the length in bytes of the memory, otherwise 0. */
	uint64_t       data_size; /* size of the "data" buffer in bytes */
	bool           can_free;  /* flag to show if the pointer values should be free'ed when the DIL_ReleaseProperty is called */
} DILProperty;
```

### DILPropertyGroup struct.

This structure contains a logical grouping of properties, this is a convenience feature to assist with providing a "pretty" command line, or GUI.

All properties are assigned to a group, and properties can only appear in one group.

```cpp
typedef struct DILPropertyGroup_t
{
  const char*  name;           /* The name for this group of properties. */
  const char*  description;    /* A description of the property group. */
  DILProperty* properties;     /* An array of properties for this group. */
  uint32_t     property_count; /* The length of the properties array. */
	bool         can_free;       /* flag to show if the properties pointer should be free'ed when the DIL_ReleaseProperty is called */
} DILPropertyGroup;
```


# NDK

For integration in TVs and STBs, V-Nova has created the LCEVC NDK (Native Development Kit) aimed at supporting the development of on-chip capabilities for decoding of MPEG-5 LCEVC.

Deploying LCEVC on the chipset using the V-Nova LCEVC NDK guarantees:

* improved content security
* increased decoding performance
* reduced power consumption

V-Nova LCEVC NDK is a collection of tools to support Video SOC vendors and ASIC designers in accelerating the availability of LCEVC on their roadmap. The toolkit includes optimised libraries that can be used at driver level as well as fully functional and compliant ASIC IP blocks. These blocks are made available as RTL source code ready for synthesis, test vectors and the relevant documentation. Furthermore, our Integration Services resources are at hand to customise the NDK for a given chipset architecture or application.


# LCEVC Decoder IP Core

V-Nova LCEVC decoder IP core is a versatile solution that supports high resolutions up to 4K, 8K, or even 16K, with frame rates 60 -120 fps. It includes optional frame buffer compression to minimize memory requirements. Through the use of AXI MM and AXI Streaming interfaces, it can easily adapt to different interface protocols. It has been compiled for and verified on Xilinx Alveo U50 and U250 boards, guaranteeing reliability and performance.&#x20;

The LCEVC Decoder Core features interfaces for efficient data transfer, communication with the host, and streaming of decoded output. By leveraging LCEVC decoder IP core, you can easily incorporate LCEVC into your next generation video SoC and take advantage of its enhanced video coding capabilities.

## **Design Features of the V-Nova LCEVC Decoder IP**

At the heart of the V-Nova LCEVC Decoder IP are distinctive design features to address the video decoding needs of different applications and SOC architectures:

**MPEG-5-part2 LCEVC Specification Compliance**

Foremost among these features is the Decoder IP's meticulous alignment with the MPEG-5-part2 LCEVC specification (ISO/IEC 23094-2). This adherence ensures complete compatibility with all defined features and profiles.&#x20;

**Adaptable Integration Architecture**

Additionally, the Decoder IP has been designed with adaptability in mind. This includes the parametrization of vital metrics such as maximum resolution, performance throughput, latency, logic area, and DDR bandwidth availability. This flexible approach enables integrators to tailor the Decoder IP's configuration to align precisely with their project's unique requirements.

**Input and Output Flexibility**

The Decoder IP has also been designed to support a multitude of base decoders with diverse characteristics. Additionally, the Decoder IP offers a runtime selection feature that allows users to choose between writing the output to DDR memory or streaming it directly to the display.

**High Performance**

The Decoder IP design targets high performance metrics to satisfy the most demanding applications. It reaches a maximum throughput of 16K resolution at 120 frames per second, accommodating 4:4:4 14-bit per pixel video formats. The design flexibility extends to the configuration of Bit-D[^1]epth, with choices between 8 or 16, as well as the internal parallelization factor Pixels-Per-Clock (PPC), offering options of 8, 16, or 32. Furthermore, the integration system clock is tailored to meet the demands of different resolutions, with 200MHz for resolutions up to 4Kp60 and 800MHz for resolutions of 8K and beyond.

## **LCEVC Decoder IP Architecture Options**

<div align="center" data-full-width="false"><figure><img src="https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/zPdCJvjNFufJbKNqq0VC/0" alt=""><figcaption><p>AXI-MM, AXI-Lite and AXI-S interfaces are provided: AXI-4 for data access and transfers, AXI-Lite for communication with the Host Controller, AXI-Stream optionally for streaming the final enhanced decoded output</p></figcaption></figure></div>

The LCEVC Decoder IP offers two setup options:

1. **Serial Processing Design (Minimum Resource Usage):** This architecture conserves resources, suiting scenarios prioritizing efficiency and resource minimization.
2. **Parallel Processing Architecture (Maximum Speed):** Designed for optimal decoding speed and latency, this architecture utilizes more resources to enhance processing performance.

**Advanced Configurations and Frame Buffer Compression:**

To minimize memory utilisation for 16K or 8K resolutions or 120 fps video, it is recommended to integrate V-Nova's frame buffer compression. This will increase logic resource utilization, requiring careful consideration.&#x20;

V-Nova Integration Services team can also work to further adapt the architecture and include proprietary frame buffer compression solutions.&#x20;

## LCEVC Decoder Integration

<figure><img src="https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/Sdd3Ti7NEl3QpurJIQTl/3" alt=""><figcaption><p>A typical integration of the LCEVC decoder core</p></figcaption></figure>

Integrating the LCEVC Decoder IP involves a series of steps:

1. **Extracting LCEVC from the Video Stream:** The process involves using a Stream Demux block to separate the input video stream into two separate elementary components: the base codec compressed frames and LCEVC enhancement frames.
2. **Storage in Frame Buffers:** These frames are then stored in dedicated frame buffers within the SDRAM.
3. **Base Decoding:** The Base decoder performs the decoding of base frames and stores the reconstructed base frames in designated Recon frame stores, also situated in SDRAM.
4. **LCEVC Read:** The LCEVC decoder core accesses both the LCEVC enhancement frames and the decoded base frames from the SDRAM frame buffers.
5. **Final Output Options:** Ultimately, the LCEVC decoder offers two output options for the decoded video frames:
   * SDRAM frame buffer.
   * Streaming output interface (if present).

\
The diagram below shows the LCEVC Decoder IP inside a typical SOC implementation:

<figure><img src="https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/vq5oBUQ0rSZ58vcoHxVd/image.png" alt=""><figcaption></figcaption></figure>

##

[^1]:


# LCEVC Decoder IP Core Deliverables

The V-Nova LCEVC Decoder IP package contains the following and can be made available on V-Nova's [Download Portal](https://download.v-nova.com/) upon request:&#x20;

* **RTL source code** ​
  * Verilog & System-Verilog code base​
* **Example C control software**​
  * Host driver code fully integrated with FFmpeg​
* **Bit accurate executable software reference model**​
  * MPEG-5 LCEVC software decoder reference model (with built-in RTL vector generation hooks)​
* **Documentation​**​
  * Block and system level specification and architectural descriptions​
* **Simulation test benches** ​
  * System Verilog behavioural test benches for RTL and gate-level verification​
* **Integration support** ​
  * Basic conformance stream set
  * Design and development support by V-Nova Integration Services team
  * Compliance and performance test suite​ (available from V-Nova partners)
  * On-going technical support​ (optional)


# LCEVC Hybrid Driver-level Decoder

The LCEVC hybrid driver-level decoder is a driver-layer solution that utilizes existing hardware blocks and optimizes CPU/GPU resources to deliver high-performance LCEVC decoding on the SoC.

## Introduction

Traditionally, video decoders are implemented either in hardware or software. Hardware solutions provide high performance but come with higher costs, while software solutions are more affordable but face performance limitations. The LCEVC Hybrid driver-level solution offers an innovative approach by leveraging existing hardware blocks, such as the Base Video Decoder, Upsample, and Blend, to handle computation-intensive tasks, while LCEVC bitstream decoding is performed in software on the device's CPU. This hybrid solution delivers an optimal balance between decoding performance and cost. Additionally, it is designed to operate at the device driver level, facilitating easy access to hardware resources.

The following diagram provides a high-level overview of how the LCEVC hybrid driver-level decoder functions.

<figure><img src="https://841975049-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FII3HaKNLFEqGvuN59zwF%2Fuploads%2Fxjs1USzQ52vRtDsSTvi3%2Fimage.png?alt=media&amp;token=70c09291-cb1c-4dfd-a965-1fd365560987" alt=""><figcaption><p>LCEVC hybrid driver-level decoder</p></figcaption></figure>

* LCEVC Extraction is the module responsible for extracting LCEVC data from the input video buffer.
* LCEVC Decoder is the module that decodes the LCEVC bitstream to generate LCEVC residual planes. It is offered as a software library by V-Nova.
* Base Video Decoder is the module responsible for decoding the base video data. It is typically available as a hardware block on the SoC.
* Upsample is the module that increases the resolution of a video frame. It is typically available as a hardware block in the SoC. The upsampled kernel data will be provided by the LCEVC Decoder.
* Blend is the module that combines multiple video layers to create a composite image or video output. It is typically available as a hardware block on the SoC.

## Features of the solution

* **High decoding performance:** Leveraging the device's hardware blocks, the driver-level decoder solution delivers exceptional decoding performance, often exceeding the capabilities of software decoders. On a quad-core ARM Cortex-A53 processor, the LCEVC decoder is capable of achieving 4K at 60 FPS.
* **Transparent application integration:** As the name implies, the LCEVC hybrid driver-level solution operates at the device driver level, ensuring transparency to application development. For example, on devices running Android or Android TV, the LCEVC hybrid driver-level solution is integrated into the SoC’s decoder implementation within the OpenMAX or C2 (Codec2) layer. Due to the seamless integration, it remains transparent to players like ExoPlayer, which is built on top of MediaCodec.
* &#x20;**Reduce time-to-market:** The hybrid approach allows LCEVC support to be added to existing SoCs through an over-the-air software update, significantly shortening the product's time-to-market cycle.


# Chromium

This section describes the V-Nova LCEVC integration into Chromium.

**Note:** The Chromium build integrated with V-Nova LCEVC is a fork from the Chromium open source project. The integration is currently still in beta and is not aligned with all versions of the main project. Please contact us for more details if you are interested in deploying with your solution.

[Chromium ](https://www.chromium.org/)is a Google-maintained open-source browser project. LCEVC Chromium is a derived work by V-Nova Ltd, that adds support for playing back streams enhanced with MPEG-5 Part 2 LCEVC using HTML5 video elements, Media Source Extension or Encrypted Media Extensions.

The official project page is <https://www.chromium.org/>. which provides information about the project, design documents, getting the code, and build guides.

### Design Overview <a href="#toc86073198" id="toc86073198"></a>

Media playback in Chromium encompasses a large array of technologies, so by necessity, this will provide only a brief outline. Inside the Chromium source media/ directory resides the components for media demuxing, software and hardware video decode, audio output, as well as audio and video rendering. ([source](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/media/README.md))

Specifically, under the playback heading, media/ contains the implementations of components required for HTML media elements and extensions:

* [HTML5 Audio & Video](https://dev.w3.org/html5/spec-author-view/video.html)
* [Media Source Extensions](https://www.w3.org/TR/media-source/)
* [Encrypted Media Extensions](https://www.w3.org/TR/encrypted-media/)

The following diagram provides a simplified overview of the media playback pipeline and where LCEVC is integrated.

![Chromium architecture with LCEVC](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/pwGhXL5U08WLE2kp1aDj/0)

As seen in the previous figure, the LCEVC integration point receives the encoded image frame (plus the enhancement data) after being demuxed by the FFmpeg demuxer (HTML video element pipeline), chunk demuxer (Media Source Extensions pipeline), and from CDM in the Encrypted Media Extensions pipeline.

The following figure shows the design of the unmodified versus LCEVC integrated video pipeline which can be applied to Chromium in the DemuxerStream object.

!\[Unmodified versus LCEVC-enhanced decoding pipelines in Chromium

]\(/files/fmwJgZu4zObYlI87qV0M)

### Supported formats <a href="#toc86073199" id="toc86073199"></a>

The V-Nova LCEVC Chromium integration adds support for;

* LCEVC H264 in an MP4 container
* LCEVC VP8 & VP9 in a WEBM container
* DRM Protected Content

### Components <a href="#toc86073200" id="toc86073200"></a>

The V-Nova LCEVC Chromium integration affects the media module in the following manner:

* media/filters/DecoderStream: Receives the encoded data and returns the decoded frame. Modified to pass LCEVC enhancement data to the LCEVC Decoder Integration Layer (DIL) and to use the DIL to generate the final video frame.
* media/lcevc/LcevcDecoderContext: handles DIL component integration.

### Usage <a href="#toc86073201" id="toc86073201"></a>

This integration is a fork from the Chromium open source project. The solution provides a way to dynamically load the LCEVC libraries. These libraries should be available in the following manner:

* Windows: the DLLs must be located in the executable directory or available through the PATH environment path
* Linux: Add the .so files location *to the LD\_LIBRARY\_PATH environment variable.*&#x20;

This integration detects the input format and adds the enhancement to every video played using the Chromium video pipeline, so no other modification is needed.

### Checkout, Build, & Run Chromium <a href="#toc86073202" id="toc86073202"></a>

In order to be able to build Chromium in your environment, you will need to follow the guidelines for your OS that can be found [here](https://www.chromium.org/developers/how-tos/get-the-code).

### Setting up the environment <a href="#toc86073203" id="toc86073203"></a>

In Windows, [Chromium ](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/windows_build_instructions.md#Setting-up-Windows)requires Visual Studio 2017 (>=15.7.2) to build, but VS2019 (>=16.0.0) is preferred and Windows SDK debug tools. Visual Studio can also be used to debug Chromium, and VS2019 is preferred for this as it handles Chromium's large debug information much better. The clang-cl compiler is used but Visual Studio's header files, libraries, and some tools are required. Visual Studio Community Edition should work if its license is appropriate for you. You must install the "Desktop development with C++" component and the "MFC/ATL support" sub-components.&#x20;

Full details of build prerequisites and instructions for Windows and Linux can be are provided by Chromium project here:

* <mark style="color:blue;">h</mark>[ttps://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows\_build\_instructions.md](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md)

&#x20;You can then continue preparing your environment by installing the depot\_tools. These are tools for working with Chromium development.

### Getting the code <a href="#toc86073204" id="toc86073204"></a>

Access to the LCEVC Chromium repository can be requested by contacting V-Nova via <info@v-nova.com>.

The code for the Chromium project can be retrieved by using the fetch tool from depot\_tools, as described in&#x20;

* <mark style="color:blue;"><https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build\\_instructions.md#Get-the-code> or</mark>&#x20;
* <mark style="color:blue;"><https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows\\_build\\_instructions.md#Get-the-code>.</mark>

Once the repository has been fetched, go into the Chromium git repository (in the `src` sub-directory), add the LCEVC-Chromium git repository as a remote, fetch and checkout its `main` branch, then sync the dependencies by using the following command:

`gclient sync -D --force --reset --with_branch_heads`

The binary libraries from the LCEVC SDK should then be copied to `media/lcevc_dependencies`, and the header files to `media/lcevc/includes`.

### Building <a href="#toc86073205" id="toc86073205"></a>

You can create any number of *build directories* with different configurations. To create a build directory:

`gn gen out/Default`

* You only have to run this once for each new build directory.
* You can replace Default with another name, but it should be a subdirectory of out.

### Enabling proprietary codecs and DRM (with Widevine) <a href="#toc86073206" id="toc86073206"></a>

In order to include support for H264, you need to enable proprietary codecs. Also, to enable DRM and EME (Encrypted Media Extensions) with [Widevine](https://www.widevine.com/) you need to add the following lines in the out/Default/args.gn file

``# Set build arguments here. See `gn help buildargs`.``

`ffmpeg_branding = "Chrome"`

`proprietary_codecs = true`

`enable_widevine = true`

### Building for Production <a href="#toc86073207" id="toc86073207"></a>

When building for Production, you should disable DCHECK as it stops Chromium execution when some validations are not met. This DCHECK is usually disabled in release distributions.

To disable DCHECK and create a Release build you should add the following lines in the out/Default/args.gn

`is_debug = false`

`dcheck_always_on = false`

### Add V-Nova DLLs to the system path in Windows <a href="#toc86073208" id="toc86073208"></a>

In order to use the V-Nova DLLs, you need to add this folder to the system path or copy all the DLLs to the chromium/src/out/Default folder.

### Add V-Nova libraries to the system path in Linux <a href="#toc86073209" id="toc86073209"></a>

In order to use the V-Nova .so files, you need to add this folder to the /usr/lib folder or add the dependency folder to the LD\_INCLUDE\_PATH.

### Build with autoninja <a href="#toc86073210" id="toc86073210"></a>

To build chromium, run the following command:

`autoninja -C out/Default chrome`

or, to produce installer packages:

`autoninja -C out/Default chrome/installer/mini_installer`

for Windows, or

&#x20;`autoninja -C out/Default chrome/installer/linux:unstable.deb`

for Debian-like Linux systems etc.

### Supported Operating Systems <a href="#toc86073211" id="toc86073211"></a>

* Windows 10
* Linux: Ubuntu 18.04, Ubuntu 20.04, Debian 10, Debian 11


# WebRTC

This section describes how MPEG-5 LCEVC enhancement can be deployed in products and solutions using WebRTC.

**Note:** The WebRTC build integrated with V-Nova LCEVC is a fork from the WebRTC open source project. The integration is currently still in beta and is not aligned with all versions of the main project. Please contact us for more details if you are interested in deploying with your solution.

With [WebRTC](https://webrtc.org/), you can add real-time communication capabilities to your application that works on top of an open standard. It enables video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions. The technology is available on all modern browsers as well as on native clients for all major platforms. The technologies behind WebRTC are implemented as an open web standard and available as regular JavaScript APIs in all major browsers. For native clients, like Android and iOS applications, a library is available that provides the same functionality. ([source](https://webrtc.org/))

The official project page is <https://webrtc.org/> which provides information about the project, design documents, getting the code, and build guides.

### Design Overview <a href="#toc86070310" id="toc86070310"></a>

The WebRTC open source project comes with software implementations of the supported codecs, VP8, VP9 and H264. But the application can provide its own additional codecs when creating a PeerConnection, injecting encoder/decoder factories as dependencies. Also, it can query the available codecs and modify them (order of preference or support, and inject them back).

### SDP - Negotiation process <a href="#toc86070311" id="toc86070311"></a>

The Session Descriptor Protocol (SDP) is used by WebRTC to negotiate the parameters used between each client in the call. Each part of the call creates an offer that lists the available codecs for sending and receiving audio and video. This list is ordered by codec preference.

With the addition of LCEVC support, several new codec names ahave been added to WebRTC to make them available in the negotiation process:

* LCEVCVP8
* LCEVCVP9
* LCEVCH264

These are included with the internal factories and are therefore available for any application using this integration.

The integration makes use of the included codecs and adds the logics to wrap them inside the enhancement process.

The following figure shows the design of the unmodified versus LCEVC-enhanced video encoding pipeline.

![Default and LCEVC-enhanced encoding pipelines in WebRTC
](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/8YPrS6FX7ZDdkKPrZ0VL/0)

The following figure shows the design of the unmodified versus LCEVC-enhanced video decoding pipeline.

![Default and LCEVC-enhanced decoding pipelines in WebRTC](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/4gReCPHOUmk4I9ZLD1lr/1)

### Supported formats <a href="#toc86070312" id="toc86070312"></a>

The V-Nova LCEVC WebRTC integration adds support for;

* LCEVC H264
* LCEVC VP8
* LCEVC VP9

### Components <a href="#toc86070313" id="toc86070313"></a>

The V-Nova LCEVC WebRTC integration affects the following modules:

* Internal decoder/encoder factory: provides the list of supported codecs, and builds the decoder/encoder of the requested type.
* LCEVC Video Encoder/Decoder: making use of the base encoder/decoders and adds the logic to use the V-Nova LCEVC EIL (Encoder Integration Layer) to create enhanced stream and DIL (Decoder Integration Layer) to decode those streams.

### Usage <a href="#toc86070314" id="toc86070314"></a>

This integration is a fork from the WebRTC open source project. The solution provides a way to statically load the LCEVC libraries. These libraries should be available in the following manner:

Windows: the DLLs must be located in the executable directory or available through the PATH environment path

Linux: Add the .so files location *to the LD\_LIBRARY\_PATH environment variable.*&#x20;

WebRTC is intended to be used as a third-party library by other applications (like a web browser), and the LCEVC-enhanced version is no different. To make use of it you need to verify that the host application is compatible with the provided enhanced version and replace the standard version of WebRTC with the modified one.

To use the LCEVC codecs, the application should set the codecs preference, by ordering the list of codecs during the SDP codec negotiation. Please note, that both sides of the communication must use a LCEVC WebRTC application to successfully negotiate the use of the new codecs.

### Supported Operating Systems <a href="#toc86070315" id="toc86070315"></a>

* Windows 10
* Linux: Ubuntu 18.04, Ubuntu 20.04


# FFmpeg

## V-Nova LCEVC Muxing on output

#### Separate Track

The `-separate_track` flag controls how LCEVC data is multiplexed in the output when using FFmpeg:<br>

* Default (`-separate_track 0`)\
  The LCEVC enhancement is embedded via SEI (Supplemental Enhancement Information) within the base video stream.
* Enabled (`-separate_track 1`)\
  LCEVC data is placed in its own video track inside the same container (e.g., a separate track within the MP4 or a separate PID in a TS). This keeps the base video and enhancement separate but still fully integrated within a single file, no additional files are generated.

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params`\
`"<parameter1>=<value1>;<parameter2>=<value2>;…"` `-separate_track 1 separate_output.ts`<br>

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264` `-base_encoder x264 -eil_params` `"<parameter1>=<value1>;<parameter2>=<value2>;…"` `-separate_track 1 separate_output.mp4`

**Example output**\
\
Using MediaInfo or VLC Codec View\
\
\
`Stream #1 Video: h264 (avc1), yuv420p(tv, bt709, progressive), 960x540 [SAR 1:1 DAR 16:9], 1495 kb/s, 24 fps.`\
`Stream #2: Video lcevc (lvc1), 1920x1080, [DAR 16:9] 513 kb/s, 24 fps.`

## V-Nova LCEVC with other base encoders

As a platform, FFmpeg can combine the V-Nova LCEVC encoder with other codec implementations supported by the V-Nova plug-in system, as illustrated in Figure 1.2. This single set of libraries is currently available with support for h.264 and HEVC codecs. Supported base encoders include x264 and x265 as well as many others (e.g. NVEnc, QSV, Xilinx NGCodec, and more). Please contact V-Nova through the [Download Portal](https://download.v-nova.com/dashboard) or via <support@v-nova.com> to request a full list of supported base encoder implementations.

![Figure 1.2 — V-Nova LCEVC encoder and decoder dataflow](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/Pl5OsYXmIH2wWfZYSJ0t/2.png)

The encoded LCEVC enhancement is added to the Supplemental Enhancement Information (SEI) of the h.264 or HEVC Network Abstraction Layer (NAL) and is transmitted as standards-compliant metadata. In this way, the video stream can be decoded by any h.264 or HEVC compatible device at the base resolution, ensuring backwards compatibility.

To accommodate both the LCEVC and the base codec components (e.g. x264), this build of FFmpeg includes support for additional command-line parameters to configure LCEVC and the base encoder.

This version currently supports 8-bit, 4:2:0 encoding. 10-bit, 4:2:2 encoding is made available as untested functionality.

This build of FFmpeg supports most of the features and file types available within the FFmpeg project. The following input and output types are supported:

| MXF (OP1a) | ts/UDP |
| ---------- | ------ |
| YUV        | mp4    |
| mp4        | webm   |
| ProRes     |        |


# FFmpeg encoder

### Overview

Most standard FFmpeg command-line options are included, as well as additional options for configuring V-Nova LCEVC.

This FFmpeg release supports LCEVC with an x264 and x265 base. Additional base codecs are available (e.g. NVEnc, libvpx, QSV, Xilinx NGcodec, etc.) and a patch for your build of FFmpeg can be provided by V-Nova upon request.

### LCEVC codec

An additional FFmpeg video codec, LCEVC, is available in this build. It is invoked by specifying the codec to enhance and the implementation of the base codec to be enhanced. Its syntax is as follows:

`-c:v lcevc_<codec> -base_encoder <codec implementation> -eil_params "<enhancement parameters string>;<base parameters string>"`

Where `<codec>` can be h264 or hevc, and `<codec implementation>` can be a specific software implementation such as x264 or nvenc\_hevc.

The behaviour of `lcevc_<codec>` is described by the following help command in FFmpeg:

`ffmpeg -help encoder=lcevc_h264`

`ffmpeg -help encoder=lcevc_hevc`

`eil_params` is a command line string that is used to pass parameters to both enhancement and base codec. Its behaviour is described later.

### Bitrate, GOP length and framerate

There are a number of parameters that are considered *generic*, i.e. not specific in use to either LCEVC or the base codec it enhances. These parameters are bitrate, GOP length and framerate. Furthermore, these parameters are used to calculate initialisation values for LCEVC and base codec.

Bitrate, GOP length and framerate are NOT set through the `eil_params` string. Instead, they use the FFmpeg generic command line options:

* -b:v for video bitrate
* -g for GOP length
* -r for framerate

Since these settings are used by the LCEVC integration layer to initialise the common rate control engine, the equivalent options in the base codec are deprecated. For example, *`bitrate=1000k`* or *`keyint=120`* may produce unexpected behaviour or return an error. Please, use *`-b:v`* and *`-g`* instead.

### EIL (Encoding Integration Layer) parameters and syntax

You can find an overview of LCEVC-specific parameters here:

{% content-ref url="/pages/QhpX63oyuRzYctfDEWu7" %}
[V-Nova LCEVC-specific Parameters](/technologies/mpeg-5.lcevc/encoder/sdk/eil/v-nova-lcevc-specific-parameters)
{% endcontent-ref %}

#### EIL parameters and syntax overview

The Encoding Integration Layer (EIL) is a V-Nova library that combines the base encoder and the LCEVC enhancement, orchestrating the combined behaviour. Moreover, the EIL is architected to support any base codec implementation through a plug-in system that makes it generic and independent. The EIL will parse the command line parameters at initialisation and configure both base and enhancement as required.

It is through the `eil_params` parameter, that both the LCEVC enhancement and the base codec are configured, for everything other than bitrate, frame rate and GOP length. Its syntax is as follows, where `eil_params` is a semicolon-separated string of parameters and values that are passed to the LCEVC enhancement layer:

`ffmpeg.exe -i input.mp4 -c:v lcevc_h264 -base_encoder x264 -eil_params "<parameter1>=<value1>;<parameter2>=<value2>;…" output.ts`

\<parameter1>, \<parameter2>, etc., are passed to the EIL interface, and configure both the V-Nova LCEVC encoder and the base encoder.

## V-Nova LCEVC FFmpeg encoder: examples of recommended commands

### 1080p and higher resolution

The following examples refer to encoding a 1080p60 YUV source.

Important: If you intend to run objective metrics, then please remember to disable dithering on the decoder.

#### CBR (with CBR base - recommended)

Example of recommended command line for 1080p CBR with CBR base, tune VQ, with bitrate set at 3 Mbps:

`ffmpeg.exe -y -framerate 59.97 -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i input-p60-1920x1080.yuv -c:v lcevc_h264 -base_encoder x264 -g 120 -b:v 3000k -eil_params "preset=medium;rc_pcrf_window_type=rolling" output-p60-1920x1080_3000kbps.ts`

#### CBR (with CRF base)

Example of a recommended command line for CBR with CRF base, with bitrate set at 3 Mbps:

`ffmpeg.exe -y -framerate 59.97 -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i input-p60-1920x1080.yuv -c:v lcevc_h264 -base_encoder x264 -g 120 -b:v 3000k -eil_params "rc_pcrf_base_rc_mode=crf;preset=medium;rc_pcrf_window_type=rolling" output-CRFbase-p60-1920x1080_3000kbps.ts`

#### pCRF (uncapped)

Example of the above command line for pCRF, with pCRF set at 27:\
(notice: -b:v 0 can also be avoided, since it is the default value of -b:v for Ffmpeg)

`ffmpeg.exe -y -framerate 59.97 -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i input-p60-1920x1080.yuv -c:v lcevc_h264 -base_encoder x264 -g 120 -b:v 0 -eil_params "rc_pcrf=27;" output-p60-1920x1080_pcrf27.ts`

#### pCRF (Capped)

Example of the above command line for pCRF, with pCRF set at 27 and cap of 5mbps. \
\
`ffmpeg.exe -y -framerate 59.97 -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i input-p60-1920x1080.yuv -c:v lcevc_h264 -base_encoder x264 -g 120 -b:v 5M -eil_params "rc_pcrf=27;rc_pcrf_min_bitrate=0;rc_pcrf_base_rc_mode=crf" output-p60-1920x1080_pcrf27.ts`

### 720p and lower resolutions

The following examples refer to encoding a 360p30 YUV source.

#### CBR (with CBR base - recommended) – 360p25

Example of recommended command line for 360p CBR, bitrate set at 170 Kbps:

`ffmpeg.exe -y -framerate 25 -f rawvideo -pix_fmt yuv420p -s 640x360 -i input-p25-640x360.yuv -c:v lcevc_h264 -base_encoder x264 -g 50 -b:v 170k -eil_params "preset=medium;rc_pcrf_window_type=rolling" output-p25-640x360_170kbps.ts`

#### CBR (with CBR base) – 540p30

Example of a recommended command line for CBR, bitrate set at 450 kbps:

`ffmpeg.exe -y -framerate 29.97 -f rawvideo -pix_fmt yuv420p -s 960x540 -i input-p30-960x540.yuv -c:v lcevc_h264 -base_encoder x264 -g 60 -b:v 450k -eil_params "preset=medium;rc_pcrf_window_type=rolling" output-p30-960x540_450kbps.ts`

More advanced settings can allow to set specific CBR rates for the base, overriding the default values.

#### pCRF (uncapped) – 720p

Example of command line for pCRF set at 30:

`ffmpeg.exe -y -framerate 29.97 -f rawvideo -pix_fmt yuv420p -s 1280x720 -i input-p30-1280x720.yuv -c:v lcevc_h264 -base_encoder x264 -g 60 -b:v 0 -eil_params "rc_pcrf=30;preset=medium;rc_pcrf_window_type=rolling" output-p30-1280x720_pcrf30.ts`

## V-Nova LCEVC with other base encoders

The LCEVC enhancement layer behaves differently depending on the performance of the base codec and plugin it is working with. Please, consult V-Nova on how to best tune supported base encoder plugins.

### x265

The LCEVC-enhanced x265 encoder can be setup as follows:

`-c:v lcevc_hevc -base_encoder x265 -eil_params "<enhancement parameter string>;”`

The following command line is an example using configurations for both the enhancement and the base codec.

`ffmpeg.exe -y -i source.y4m -c:v lcevc_hevc -base_encoder x265 -g 60 -b:v 1000k -eil_params "preset=medium;lcevc_tune=vq;" output.ts`

Note that GOP Length is no longer passed directly to x.265 using keyint as a parameter in the eil\_params string. Rather, it is set using the global -g option in FFmpeg. Other options for GOP length, such as min-keyint remain available and applicable with the use of scene-cut.&#x20;

### **AV1 (LibAOM)**

The LibAOM encoder is configured using the formatting below:

`-c:v lcevc_av1 -base_encoder aom -eil_params "<enhancement parameter string>;”`

Here is a full example command, encoding AV1 with LCEVC in FFmpeg:

`ffmpeg -y -i input.yuv -c:v lcevc_av1 -base_encoder aom -g 60 -b:v 1000k -eil_params "threads=8;cpu-used=4;row-mt=1;lcevc_tune=vq;" output.webm`

To define the balance of encoding speed and quality libaom uses 'cpu-used' with a range 0-8 (0 Quality, 8 performance) . This should be used in partnership with [`lcevc_preset`](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#lcevc-preset) guidance.

Note that B-Frames are not applicable in AV1, the picture type structure is different and uses S-frame and Intra frames. There is no parameter control for frame types but the -g (GOP) setting is critical to define the interval of the 'Intra' frame type.

Note that the output container needs to be webm.

### AV1 (SVT-AV1)

Intel's SVT-AV1 is configured using the formatting below:

`-c:v lcevc_av1 -base_encoder svt_av1 -eil_params "<enhancement parameter string>;”`

'Preset' is the control to define the balance of encoding speed over quality for the base codec. The values available are 0-13 (P1 quality, P12 performance). This should be used in partnership with [`lcevc_preset`](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#lcevc-preset) guidance.\
\
Note that B-Frames are not applicable in AV1, the picture type structure is different and uses S-frame and Intra frames. There is no parameter control for frame types but the -g (GOP) setting is critical to define the interval of the 'Intra' frame type.

Note that the output container needs to be webm

### **VP9**

VP9 is configured in a similar way to x265 where most extra parameters are required within the eil param string otherwise they won't be passed corretly:

`-c:v lcevc_vp9 -base_encoder vpx_vp9 -eil_params "<enhancement parameter string>;”`

The following command line is an example using configurations for both the enhancement and the base codec.:

`ffmpeg -y -i source.yuv -c:v lcevc_vp9 -base_encoder vpx_vp9 -s 1920x1080 -r 30 -b:v 3000k -g 120  -eil_params "quality=good;cpu-used=0;row-mt=1" output.webm`

To define the balance of encoding speed and quality VP9 uses; 'cpu-used' *(0-8)* & 'quality' *(Best, good, realtime).*\
When the quality parameter is `good` or `best`, values for `-cpu-used` can be set between 0 and 5, when quality is set to `realtime`, the available values for `-cpu-used` are 0 to 8 (0 Quality, 8 performance). This should be used in partnership with [`lcevc_preset`](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#lcevc-preset) guidance.&#x20;

As guided in the [VP9 documentation](https://developers.google.com/media/vp9) `-row-mt=1;`is advised to enable multi threading operation.&#x20;

Note B-Frames are not applicable in VP9, the picture type structure uses Intra or Key frames. There is no parameter control for frame types but the -g (GOP) setting is critical to define the interval of the 'Intra' frame type.

Note that the output container needs to be webm.

### QSV

Quick Sync Video is Intel's hardware dedicated encoding and decoding tools, the codec has certain hardware requirements which can be found on [Intel's website](https://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html). QSV supports encoding of H264, HEVC, VP8 and VP9. Encoding with LCEVC is similar to previous examples:

`-c:v lcevc_h264 -base_encoder qsv_h264 -eil_params "<enhancement parameter string>;”`

`-c:v lcevc_hevc -base_encoder qsv_hevc -eil_params "<enhancement parameter string>;”`

The following command line is an example using configurations for both the enhancement and the base codec:

`ffmpeg -y -vcodec rawvideo -pix_fmt yuv420p -s 1920x1080 -r 30 -i source.yuv -c:v lcevc_hevc -s 1920x1080 -r 30 -b:v 5000k -g 60 -base_encoder qsv_hevc output.ts`

There is different level of support for MFX (aka Intel Media SDK) between Windows and Linux on Windows - supported on a broader range of chipsets on Linux - supported on gen8+ chipsets only, which use iHD driver for older chipsets on Linux a direct integration of VAAPI is required, basically an extra EIL plugin, how difficult - unknown. This will enable older Intel chipsets to be used on Linux for LCEVC encodes (with hardware base encoding, which base codecs are supported by which generations of chipsets - another question

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/HEjyfEEFSecMgVAB6gaX/Selection_155.png)

### NVENC Plugin

NVENC is NVidia's built-in video encoding available on a broad range of its GPU cards, details on the specific support from different NVidia hardware is [available here](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new). NVENC can support both AVC/H.264 and HEVC/H.265 encoding depending upon the GPU in question. Running LCEVC-enhanced encoding of these two base encoders can be initiated with the following commands:

**CPU** - The LCEVC encoding is performed solely on the CPU with the NVIDIA GPU's NVENC hardware encoder providing the base layer encodin

`-c:v lcevc_h264 -base_encoder nvenc_h264 -eil_params "<enhancement parameter string>;”`

`-c:v lcevc_hevc -base_encoder nvenc_hevc -eil_params "<enhancement parameter string>;”`

**Vulkan** - This process primarily uses the GPU to encode the LCEVC enhancement.&#x20;

`-c:v lcevc_h264_vulkan -base_encoder nvenc_h264 -eil_params "gpu_device=NVIDIA”`

`-c:v lcevc_hevc_vulkan -base_encoder nvenc_hevc -eil_params "gpu_device=NVIDIA”`

Within FFmpeg, NVidia is not able to access frame data in normal system memory. All frame data needs to be uploaded to hardware surfaces connected to the appropriate device before being used. As per Nvidia documentation [here](https://docs.nvidia.com/video-technologies/video-codec-sdk/12.0/ffmpeg-with-nvidia-gpu/index.html). \
Within FFmpeg the following example creates a filter where rawvideo is uploaded in hardware, then passed to the stream named.

`-filter_complex '[0]hwupload[input_in_gpu_memory]' -map '[input_in_gpu_memory]'` &#x20;

Following the same logic as other base encoders, bitrate, frame rate, resolution and GOP are specified as normal in FFmpeg, however other commands need to be within the eil\_params.\
\
Post NVENC SDK 10.0 the API structure changed. For a guide to translate your preset please see [here](https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-preset-migration-guide/). Old preset options; \[default, hp, hq, lossless, lossless\_hp] are now updated to be an integer with 'preset=X' where the value range is 1-7 (P7 highest quality, P1 performance).


# FFmpeg decoder

## V-Nova LCEVC FFmpeg decoder commands

The V-Nova LCEVC decoder can be used similarly to other codecs in FFmpeg or FFplay, with one important difference. The enhancement layer is embedded as metadata in a fully compliant and backward compatible H.264/AVC or H.265/HEVC elementary stream. Therefore, the decoder must be instructed to extract and decode LCEVC data.

The command line to enable V-Nova LCEVC decoding of an LCEVC-enhanced transport stream or MP4 file, is as follows:

`ffmpeg.exe -vcodec lcevc_h264 -i stream.ts -vcodec rawvideo output.yuv`

`ffmpeg.exe -vcodec lcevc_h264 -i stream.mp4 -vcodec rawvideo output.yuv`

The equivalent command line for HEVC is:

`ffmpeg.exe -vcodec lcevc_hevc -i stream.ts -vcodec rawvideo output.yuv`

`ffmpeg.exe -vcodec lcevc_hevc -i stream.mp4 -vcodec rawvideo output.yuv`

And for FFplay, h.264 and HEVC respectively:

`ffplay.exe -vcodec lcevc_h264 -i stream.mp4`

`ffplay.exe -vcodec lcevc_hevc -i stream.mp4`

### Decoding to perform Objective evaluation

To perform objective metric calculations when dithering is enabled in LCEVC encoding, dithering **must** be disabled as follows:

`ffmpeg.exe -vcodec lcevc_h264 -disable_dithering 1 -i stream.mp4 -vcodec rawvideo output.yuv`

To calculate metrics, the command line is similar to that of other codecs in FFmpeg, in line with guidelines included at the following link, and as shown in the following command line:

<https://github.com/Netflix/vmaf/blob/master/resource/doc/libvmaf.md>

`ffmpeg.exe -vcodec lcevc_h264 –i input_stream.ts -vcodec rawvideo -s 1920x1080 -i reference_yuv.yuv -filter_complex " [0:v]scale=1920x1080:flags=bicubic[main];[main][1:v]libvmaf" -f null`


# Putting the software together

The LCEVC-enabled FFmpeg build can be easily assembled. You will have received software in the following packages:

* FFmpeg: the FFmpeg binaries with support for x264 and x265
* LCEVC:  the LCEVC libraries both encoder and decoder libraries
* base codecs:  any additional base codecs requested (if applicable)

To set it all up, simply:

1. UnZip the FFmpeg binaries for your operating system into a local directory of your choice (or create a new one). This will be your “FFmpeg directory”.
2. UnZip the LCEVC encoder and decoder package and Copy the LCEVC encoder and decoder libraries with their subfolders onto the root  FFmpeg directory. Make sure the dynamic libraries of LCEVC encoder and decoder are put in the root FFmpeg directory.
3. Copy any additional base codecs and base codec plug-ins that you may have received from their folder onto the root FFmpeg directory (if applicable).

In some cases, for example when installing the LCEVC-enabled FFmpeg build on a clean docker container, you may need to install some or all of the dependencies listed below. Alternatively, installing the generic FFmpeg build may include some of these dependencies.

`nasm pkg-config libxml2-dev`&#x20;

Other potential dependencies for an LCEVC-enabled FFmpeg build may include: &#x20;

`dbus libapparmor1 libasound2 libasound2-data libasyncns0 libbsd0 libc6 libdbus-1-3 libexpat1 libflac8 libfontconfig1 libfontenc1 libfreetype6 libgl1 libglvnd0 libglx0 libice6 libogg0 libpng16-16 libpulse0 libsdl2-2.0-0 libsm6 libsndfile1 libuuid1 libvorbis0a libvorbisenc2 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwrap0 libx11-6 libx11-xcb1 libxau6 libxaw7 libxcb-dri3-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcb-util1 libxcb-xfixes0 libxcb-xinerama0 libxcb-xkb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxdmcp6 libxext6 libxfixes3 libxft2 libxi6 libxinerama1 libxkbcommon0 libxkbfile1 libxmu6 libxmuu1 libxpm4 libxrandr2 libxrender1 libxres1 libxss1 libxt6 libxtst6 libxv1 libxvmc1 libxxf86vm1 xkb-data zlib1g`

Quick Tip:  For Ubuntu, running `sudo apt-get install -y`  followed by the above dependencies list will install all of them at once.

On Linux systems, especially where more than one build of FFmpeg is available, you may have to set the library path to the local folder for the command to work:

`sudo LD_LIBRARY_PATH=.`

For further support or to report issues, please reach out to V-Nova through the [Download Portal](https://download.v-nova.com/dashboard) or via <support@v-nova.com>.&#x20;

**Please note, LCEVC is proprietary to V-Nova and subject to V-Nova's proprietary licence. Therefore, distribution of any pre-compiled subsystem is strictly prohibited, even between group companies.**

#### **Additional base codecs**

LCEVC can enhance any codec implementation through a simple plug-in system. V-Nova has developed multiple plug-ins for the most popular base codec implementations. These can be requested and will be packaged as part of a release in a separate **base codecs** folder. The folder will include:

* the base codec **plug-ins**
* (optionally) the base codec **libraries**

The content of the above folders needs to be copied into the root FFmpeg directory


# Example script

to encode, decode and calculate metrics

{% hint style="warning" %}
Important: If you intend to run objective metrics, then please remember to disable dithering on the decoder.
{% endhint %}

`~/vnova/ffmpeg/ffmpeg \`\
&#x20;`-framerate $FPS \`\
&#x20;`-vcodec rawvideo \`\
&#x20;`-pix_fmt yuv420p \`\
&#x20;`-s $RESO \`\
&#x20;`-i $INPUT \`\
&#x20;`-vcodec lcevc_h264 \`\
&#x20;`-base_encoder x264 \`\
&#x20;`-b:v $BITRATE \`\
&#x20;`-eil_params "lcevc_tune=vmaf;dc_dithering_type=none;preset=medium;rc_pcrf_window_type=rolling" \`\
&#x20;`-f mp4 \`\
&#x20;`outputs/"$FILE"_vnova.mp4`

`ffmpeg \`\
&#x20;`-s $RESO \`\
&#x20;`-framerate $FPS \`\
&#x20;`-vcodec rawvideo \`\
&#x20;`-i $INPUT \`\
&#x20;`-vcodec libx264 \`\
&#x20;`-b:v $BITRATE \`\
&#x20;`-preset medium \`\
&#x20;`-g $FPS \`\
&#x20;`-f mp4 \`\
&#x20;`outputs/"$FILE"_libx.mp4`

`~/vnova/ffmpeg/ffmpeg -y \`\
&#x20;`-vcodec lcevc_hevc \`\
&#x20;`-disable_dithering 1 \`\
&#x20;`-i outputs/"$FILE"_vnova.mp4 \`\
&#x20;`-vcodec rawvideo \`\
&#x20;`-s $RESO \`\
&#x20;`-framerate $FPS \`\
&#x20;`-pix_fmt yuv420p \`\
&#x20;`-i $INPUT \`\
&#x20;`-filter_complex "[0:v]scale=1920x1080:flags=bicubic[main];[main][1:v]libvmaf=model_path=/path/to/vmaf_v0.6.1.pk1" \`\
&#x20;`-f null -`&#x20;

`~/vnova/ffmpeg/ffmpeg -y \`\
&#x20;`-i outputs/"$FILE"_libx.mp4 \`\
&#x20;`-s $RESO \`\
&#x20;`-framerate $FPS \`\
&#x20;`-pix_fmt yuv420p \`\
&#x20;`-i $INPUT \`\
&#x20;`-filter_complex "[0:v]scale=1920x1080:flags=bicubic[main];[main][1:v]libvmaf" \`\
&#x20;`-f null -`


# Player integrations


# AVPlayer

### Introduction <a href="#toc75867822" id="toc75867822"></a>

This page provides a guide to adding LCEVC-enhanced video decoding capabilities to an iOS application based on the native AVPlayer.

### Including the LCEVC Libraries <a href="#toc75867823" id="toc75867823"></a>

As a first step you need to add the provided LCEVC libraries and dependencies to your project. If you don't already have access to the [LCEVCAVPlayer Github repository](https://github.com/v-novaltd/LCEVCAVPlayerKit) please speak to your V-Nova representative or submit a request [here](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/). There are several possible approaches to this depending on your preference:

#### CocoaPods

[CocoaPods](https://cocoapods.org) is a dependency manager for Cocoa projects. For usage and installation instructions, visit their website. To integrate `LCEVCAVPlayerKit` into your Xcode project using CocoaPods, specify it in your `Podfile`:

```ruby
pod "LCEVCAVPlayerKit", :source => 'https://github.com/v-novaltd/LCEVCAVPlayerKit'
```

#### Swift Package Manager

The [Swift Package Manager](https://swift.org/package-manager/) is a tool for automating the distribution of Swift code and is integrated into the `swift` compiler.

Once you have your Swift package set up, adding `LCEVCAVPlayerKit` as a dependency is as easy as adding it to the `dependencies` value of your `Package.swift.` It can also be added via Xcode: [more info here](https://developer.apple.com/documentation/swift_packages/adding_package_dependencies_to_your_app)

```swift
dependencies: [
    .package(url: "https://github.com/v-novaltd/LCEVCAVPlayerKit.git", .upToNextMajor(from: "1.0.025"))
]
```

#### Manually

If you prefer not to use any of the aforementioned dependency managers, you can integrate LCEVCAVPlayerKit into your project manually.

**Embedded Framework**

```swift
~$ git clone https://github.com/v-novaltd/LCEVCAVPlayerKit
```

* Open the new `LCEVCAVPlayerKit` folder, and drag the `LCEVCAVPlayerKit.xcframework` & `VNova-ffmpeg.xcframework` into the Project Navigator of your application's Xcode project. When it asks to copy, please check both files to copy them under your project folder.
* These universal frameworks (xcframework) are dynamic frameworks and have to be embedded and shipped within your app, otherwilse run time error will happen.

### Integration <a href="#toc75867824" id="toc75867824"></a>

There are two options when it comes to integrating the LCEVC-enhanced player. One provides an out of the box solution but does not allow customization, while the other requires a few extra steps but enables you to customize the player, buttons, look and feel and behavior.

### LCEVC Enhancement Pipeline <a href="#toc75867825" id="toc75867825"></a>

It is important to keep in mind how the enhancement is applied to the base video. The LCEVC encoded video includes extra information that the LCEVC enhancer can use to improve the quality and resolution of the image once it is rendered.

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/mIn2WN8o6gaimbjNASGM/2)

The result of the enhancement process is a new frame to be displayed. The rendering of the frames is handled by the LCEVCAVPlayerKit libraries which provide a custom UI View to be used to render the video.

#### Integration Key Components

The LCEVC process has been encapsulated inside a single class to facilitate a simple and rapid integration:

* **LCEVCAVPlayer:** A subtype of **AVPlayer** that adds the required interceptions and processes to generate the enhanced video. This class is compatible with the standard components that make use of AVPlayer and adds:
  * **outputVideoView:** A read-only property that allows at any point to get a reference to the view containing the enhanced version of the video. This view is maintained and created by the player and it can be easily integrated into any existing view.
  * **renderIn:** This function handles the initialization and configuration of the outputVideoView as a sub-View of the provided parent view, creating the required constraints for it to fill its parent. If the client application has custom layout management this call can be skipped.
  * **width:** A read-only property returning the width of the enhanced video. Normally this property is only useful for debugging purposes, but it can be essential if the client application needs to manipulate the layout of the application.
  * **height:** A read-only property returning the width of the enhanced video. Normally this property is only useful for debugging purposes, but it can be essential if the client application needs to manipulate the layout of the application.

### Integration QuickStart 1: Using AVPlayerViewController <a href="#toc75867826" id="toc75867826"></a>

The LCEVCAVPlayerKit libraries are compatible with apps that use the native iOS Player controls provided by AVFoundation using the AVPlayerViewcontroller. The integration itself has been simplified as much as possible to ensure that minimal effort is required.

#### Instructions

1\. import the frameworks (AVPlayerViewController needs Apple's AVKit framework)

```
import AVKit
import LCEVCAVPlayerKit
```

2\. Create an instance of LCEVCAVPlayer providing the remote URL of the asset you want to play. This is a subclass of AVPlayer and can be used in any place you would normally use your AVPlayer instance.

```swift
let player = LCEVCAVPlayer(url: url)
```

3\. Once that instance is created you can provide it to the standard AVPlayerViewController using its player property and call the **renderIn** method of LCEVCAVPlayer which will render the enhanced video to the superlayer you provide.

```swift
let view = AVPlayerViewController()
view.player = player
self.present(view, animated: true, completion: nil)
player.renderIn(view.contentOverlayView!)
```

The end result will be that the LCEVC-enhanced video will be displayed with the native iOS player controls provided by AVFoundation.

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/hZccAWMsMOvyrrOb1V5S/3)

### Integration QuickStart 2: Using LCEVCAVPlayer directly <a href="#toc75867828" id="toc75867828"></a>

This option is suitable if you need more control over the playback UI controls or further customization of the player.

#### Instructions <a href="#toc75867830" id="toc75867830"></a>

1. Add an instance of **UIView** into your application. This view will end up holding the enhanced video. This can be done either programmatically or using storyboards as below:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/rgTlIEenjs8qEZYAHPit/4)

1\. Import the framework

```
import LCEVCAVPlayerKit
```

2\. In your **ViewController**, create and maintain an instance of **LCEVCAVPlayer** providing the URL of the remote stream to be played. This player exposes all the capabilities of the standard AVPlayer with some exceptions that are outlined in the following section of this document.

```swift
let player = LCEVCAVPlayer(url: url)
```

3\. Once your view has been created, for example when the **viewDidLoad** trigger gets called, you can call the **addObserver** function on the player to monitor and react to the different status changes by using the key path **AVPlayer.status**:

```swift
player.addObserver(self, forKeyPath: #keyPath(AVPlayer.status), options: [.old, .new], context: &playerItemContext)
```

This will fire a callback for every player status change.

You should also provide the **LCEVCAVPlayer** with the required rendering target at this point.

```swift
player.renderIn(targetView)
```

The targetView argument refers to the **UIView** where the **LCEVCAVPlayer** will insert the output rendering target. It will be configured to stretch to fill its parent. In case that you want to manage the layout manually you can access that view using the **outputVideoView** property.

1. Once those steps are completed the player is configured and ready to go, the only thing left is to trigger the playback. A good place for that is to use the observer previously defined by catching the status **readyToPlay** where it is safe to use the default **play** method of the **AVPlayer** to start the playback session.

### Other Notes and Considerations

* LCEVC AVPlayer makes use of a local web server to intercept requests performed by AVFoundation. This approach is secure because the included server is configured to reject any request that does not originate from the local device.
* The current version of the enhancer internally intercepts and modifies the requested URLs in order to extract the LCEVC enhancement data. If you are currently handling your AVAssets manually (by preloading or storing them in any way) some customization may be required.
* LCEVC AVPlayer does not support AirPlay. However, mirror mode can be used.
* LCEVC AVPlayer does not support FairPlay protected content.


# VLCKit

## 1 Introduction

This section provides an overview of how to build and use VLCKit, how VLCKit uses Apple’s VideoToolbox and how LCEVC decoding can be achieved inside VLCKit.

A general overview of VLCKit and its capabilities is available at <https://wiki.videolan.org/VLCKit>.

The LCEVC-enabled VLCKit has been extensively tested and validated by third party test house Kineton to ensure that the addition of LCEVC to the player maintained all of its functionality and performance. The results of the tests are available [here](https://www.v-nova.com/wp-content/uploads/dlm_uploads/Validation_and_Certification_for_LCEVC_on_Mobile_Devices_V-Nova_Test_Report.pdf).

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/RAH90AhmZpvIZOkxnGbt/Logo-Kineton-1-wide.png)

## 2 Integrating VLCKit

The VLCKit repository can be found in [VideoLan’s Gitlab](https://code.videolan.org/videolan/VLCKit). The vanilla VLCKit is available through **CocoaPods**, **Carthage** and you can **build it locally** on your machine. The VLCKit that is commercially used is not on master branch yet, it’s stored in [branch 3.0](https://code.videolan.org/videolan/VLCKit/-/tree/3.0). For the purposes of this document, we will use the 3.0 branch and build it locally on the machine.

### 2-1 Building VLCKit

Before starting the build, make sure you have enough storage space (\~6GB).

#### 2-1-1 Vanilla VLCKit

By going into the root directory of VLCKit, you can start the build process with the following commands:

```bash
./buildMobileVLCKit.sh -f
```

You can also specify which architecture you want to build VLCKit for:

```bash
./buildMobileVLCKit.sh -a aarch64
```

Once the build finishes, you can find the **MobileVLCKit.framework** in the **build** directory. You can link this framework directly inside XCode:

1. Go to **Project > Targets > ${YOUR\_APPLICATION} > General** and add framework to **Frameworks, Libraries and Embedded content** section.
2. Make sure **Framework Search Paths** and **Header Search Paths** in **Project > Targets > ${YOUR\_APPLICATION} > Build Settings** can find the *.***framework** and its **Headers** directory.

For more information on how to integrate frameworks, check [Apple’s documentations](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Tasks/IncludingFrameworks.html).

#### 2-1-2 VLCKit + LCEVC

VLCKit has a mechanism for automatically applying patches to libvlc (VLC’s core) before starting the build. The required changes for applying LCEVC to VLCKit consist of multiple patches and a static library binary. You just need to copy the patches into **Resources/MobileVLCKit/patches** folder and start the build. Once the build is finished, integrate the framework just like before.

You also need to embed the **lcevc\_dec\_videotoolbox.a** binary into your application. Same as frameworks:

1. Go to **Project > Targets > ${YOUR\_APPLICATION} > General** and add framework to **Frameworks, Libraries and Embedded content** section.
2. Make sure **Library Search Paths** in **Project > Targets > ${YOUR\_APPLICATION} > Build Settings** can find the **lcevc\_dec\_videotoolbox.a** file.

VLCKit should now be able to decode LCEVC streams.

### 2-2 Using VLCKit

In order to use VLCKit, first you need an instance of `VLCMediaPlayer`. Then an `UIView` to set as render target and a `VLCMedia` instance which will load the asset.

{% code title="PlayerViewController.swift" %}

```swift
class PlayerViewController: UIViewController {
    var url: URL!
    let player = VLCMediaPlayer()
    
    override func viewDidLoad() {
        super.viewDidLoad()
        setupMediaLibrary()
    }
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        player.play()
    }
    
    private func setupMediaLibrary() {
        player.delegate = self
        player.drawable = view
        player.media = VLCMedia(url: url)
    }
}

extension PlayerViewController: VLCMediaPlayerDelegate {
    func mediaPlayerStateChanged(_ aNotification: Notification!) {
        if player.state == .stopped {
            self.dismiss(animated: true, completion: nil)
        }
    }
}
```

{% endcode %}

For more information on how to use VLCKit you can check their [examples](https://code.videolan.org/videolan/VLCKit/-/tree/master/Examples), [documentations](https://code.videolan.org/videolan/VLCKit/-/tree/master/doc/html) and [VLC-iOS](https://code.videolan.org/videolan/vlc-ios/).

## 3 [Apple’s VideoToolbox](https://developer.apple.com/documentation/videotoolbox)

In order to work with Apple’s VideoToolbox Decoder module (`VTDecompressionSession`), the stream must be demuxed into `CMSampleBuffer`s and fed into `VTDecompressionSession` for decompression. In response `VTDecompressionSession` will return `CVPixelBuffer`s for given `CMSampleBuffer`s in decoding order. After reordering `CVPixelBuffer`s to presentation order, they’re ready for presentation on screen.

### 3-1 VLCKit videotoolbox module

All the logics mentioned earlier are encapsulated into a single module named videtoolbox in VLCs ecosystem. The process is shown below:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/0FuDwHnR0STZLhxEhjHb/1.png)

### 3-2 VLCKit + LCEVC

In order to apply LCEVC decoding, we need to extract LCEVC payload before decompression, and apply LCEVC once decompression is done and frames are ordered by presentation. The process is shown below:

![](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/XY6WBdxPuoMsvHarBxrt/2.png)

1.


# hls.js

The page outlines the steps needed to integrate the LCEVC decoder modifications to a player based on hls.js. V-Nova provides a reference integration with hls.js which can be referred to as you update your own hls.js player deployment.

### Reference integration package

The reference integration packages contains a zip file with the following :&#x20;

1. **js/libs :** These are all the files related to the player (hls.js).
2. **js/VNova-LCEVC :** All DIL.js Libraries need for LCEVC Enhancements. (dil.min.js, liblcevc\_dpi.wasm) and patch file based on player to bridge the api calls to the player library(lcevc\_dil\_patch\_hls.js).
3. **js/app.js :** This is the UI based operations with the demo UI that comes with the package.
4. **index.html :** Default demo page that showcases the decoding capabilities with LCEVC manifests in HLS and DASH based on the player, and provides a field to specify different stream manifest links for testing.
5. **simple.html :** The bare minimum integration that handles everything inside the patch.
6. **integration.html :** This explains a basic integration with a video, canvas and config passed to the dil libraries.

### How to integrate LCEVC into an existing hls.js player

**Step 1:** To integrate LCEVC into an existing hls.js player, you need to include all the scripts under the "demo/js/VNova-LCEVC" folder.

Your HTML page should include the following scripts in the header:

```markup
<!-- a version of hls.js -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/hls.js/1.0.4/hls.js"></script>

<!-- vnova lcevc -->
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil.min.js"></script>
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil_patch_hls.js"></script>
```

Notes:

* A version of "hls.js" will need to be included. Currently, version **1.0.4** is supported.
* The file "lcevc\_dil\_patch\_hls.js" should be included last, as this needs the other scripts to be loaded first.
* All the VNova-LCEVC scripts must be present in the same folder, including "liblcevc\_dpi.wasm".

Or it is also possible to reference these sources hosted in the AWS cloud by V-Nova. These are updated periodically to the latest versions:

```markup
<script type="text/javascript" src="https://d3mfda3gpj3dw1.cloudfront.net/LCEVC_HTML/dil.js/lcevc_dil.min.js"></script>
<script type="text/javascript" src="https://d3mfda3gpj3dw1.cloudfront.net/LCEVC_HTML/dil.js/lcevC_dil_patch_hls.js"></script>
```

**Step 2:** In the HTML, the code used to load the hls.js player must be modified slightly from its normal usage, to integrate the LCEVC enhancement decoding capabilities, as follows:

**Code Changes**

To check if hls.js is supported or not:

```javascript
if (Hls.isSupported()) {
	if (hls) {
		hls.destroy();
		hls = null;
	}
} else {
	console.log('Playback is not supported');
}
```

To create a new instantiation of hls.js:

```javascript
hls = new Hls({
	debug:             false,
	enableWorker:      true,
	defaultAudioCodec: null
});
```

To initialise LCEVC within the hls player:

```javascript
var canvas = document.createElement('canvas');
var lcevcDilConfig = {
    // optional settings here
};

hls.attachLcevc(canvas, lcevcDilConfig);
```

The parameters are defined as follows:

**canvas**

This is an HTMLCanvasElement object and is used to display the rendered frames / output of the video after V-Nova LCEVC has been applied.

Its visibility, and pixel size are both controlled automatically by the player. Please note that the dimensions (in pixels) are set based on the size of the initial video and of the LCEVC-enhanced content being rendered. It is not guaranteed to always be a fixed size and may change dynamically.

The canvas element should mimic the screen size and page position of the video element, via the user’s CSS pixel size, however, this should not be forced, as it is controlled by the player.

The canvas element will be a WebGL canvas, and therefore it is not possible to use any of the HTML5 2D drawing context methods.

**lcevcDilConfig**

This is an object block containing settings with which V-Nova LCEVC is initialised. It is automatically configured with optimal settings.

Then just attach the video element as normal:

```javascript
var video = document.getElementById('video');
hls.attachMedia(video);
```

The **video** variable is an HTMLVideoElement object and contains the video, exactly as it would for the regular use of hls.js. Once V-Nova LCEVC has been loaded from the header data, this video element is automatically made invisible, using dynamic CSS. A canvas element is then used to display the rendered frames.

Autoplay when loaded:

```javascript
hls.on(Hls.Events.BUFFER_CREATED, function(event, data) {
	window.setTimeout(function() {
		video.play();
	}, 500);
});
```

##

##

```javascript
// Integration lines of code.
if(
  lcevcDil &&
  (segment.type === 'video' || segment.type === 'audiovideo')
) {
  lcevcDil.appendBuffer(
    segment.data.buffer,
    segment.type,
    loadLevel,
    timestampOffset,  // Optional
  );
}
```

#### LcevcDil.appendBuffer(data, type, level, timestampOffset)

* data: The video buffer data.
* type: The type of the data, can be `'video'`, `'audio'` or `'audiovideo'`.
* level: The current ABR profile/rendition/level ID of the data.
* timestampOffset (optional): The offset that needs to be applied to all timestamps in the buffer.


# Shaka Player

## Shaka Player version 4.3 and after

LCEVC decoding is supported in Google’s web player project [Shaka Player](https://github.com/shaka-project/shaka-player/) since version 4.3.

By visiting [shaka-player-demo](https://shaka-player-demo.appspot.com/demo/#audiolang=en-GB;textlang=en-GB;lcevc.enabled=true;drawLogo=true;uilang=en-GB;panel=ALL_CONTENT;panelData=LCEVC;build=uncompiled), you can easily try out LCEVC stream playback by enabling the following configurations:

<figure><img src="https://841975049-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FII3HaKNLFEqGvuN59zwF%2Fuploads%2FPHFJBwchP2wTrzn3Z4tf%2Fimage.png?alt=media&amp;token=9134bcee-93c9-4746-8370-6d12bda169b4" alt=""><figcaption></figcaption></figure>

How to enable LCEVC in Shaka Player has been documented in the official project github, here:&#x20;

<https://github.com/shaka-project/shaka-player/blob/main/docs/design/current/lcevc-integration.md>

## Shaka Player legacy versions (before 4.3)

You can also add LCEVC support to an older version of Shaka Player by adding a patch, which you can download from our portal: <https://download.v-nova.com/>.&#x20;

The remainder of this page describes the steps required to integrate the LCEVC decoder modifications..

### Reference integration package

The reference integration packages contains a zip file with the following :&#x20;

1. **js/libs :** These are all the files related to the player Eg: hls.js , shaka\_player.js.
2. **js/VNova-LCEVC :** All DIL.js Libraries need for LCEVC Enhancements. (dil.min.js, liblcevc\_dpi.wasm) and patch file based on player to bridge the api calls to the player library.
3. **js/app.js :** This is the UI based operations with the demo UI that comes with the package.
4. **index.html :** Default demo page that showcases the decoding capabilities with LCEVC manifests in HLS and DASH based on the player, and provides a field to specify different stream manifest links for testing.
5. **simple.html :** The bare minimum integration that handles everything inside the patch.
6. **integration.html :** This explains a basic integration with a video, canvas and config passed to the dil libraries.

### How to integrate LCEVC into an existing Shaka player

**Step 1:** To integrate LCEVC into an existing Shaka player, you need to include all the scripts under the "demo/js/VNova-LCEVC" folder.

Your HTML page should include the following scripts in the header:

```markup
<!-- a version mux js for TS compatibility (optional) -->
<script src="https://cdn.jsdelivr.net/npm/mux.js@5.9.1/dist/mux.min.js"></script>
<!-- a version of shaka player -->
<script type="text/javascript" src="js/shaka/shaka-player.compiled.debug.js"></script>

<!-- vnova lcevc -->
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil.min.js"></script>
<script type="text/javascript" src="js/VNova-LCEVC/lcevc_dil_patch_shaka.js"></script>
```

Notes:

* A version of "Shaka player" will need to be included.
* The file "lcevc\_dil\_patch\_shaka.js" should be included last, as this needs the other scripts to be loaded first.
* All the VNova-LCEVC scripts must be present in the same folder, including "liblcevc\_dpi.wasm".

**Step 2:** In the HTML, the code used to load the shaka player must be modified slightly from its normal usage, to integrate the LCEVC enhancement decoding capabilities, as follows:

**Code Changes**

To check if shaka is supported or not and install polyfills:

```javascript
shaka.polyfill.installAll();

if (!shaka.Player.isBrowserSupported()) {
  if (player) {
        if (player.lcevcDil) {
          player.lcevcDil.close();
        }
        player.destroy();
        player = null;
      }
} else {
	console.log('Playback is not supported');
}
```

To create a new instantiation of shaka player:

```javascript
var video = document.createElement('video');
// create a new shaka player instance
player = new shaka.Player(video);
// force safari on mac and ipad to use Media Source Extensions (MSE). (Optional)
player.configure('streaming.useNativeHlsOnSafari', false);
```

To initialise LCEVC within the hls player:

```javascript
var canvas = document.createElement('canvas');
var lcevcDilConfig = {
    // optional settings here
};

player.attachLcevc(video, canvas, lcevcDilConfig);
```

The parameters are defined as follows:

**canvas**

This is an HTMLCanvasElement object and is used to display the rendered frames / output of the video after V-Nova LCEVC has been applied.

Its visibility, and pixel size are both controlled automatically by the player. Please note that the dimensions (in pixels) are set based on the size of the initial video and of the LCEVC-enhanced content being rendered. It is not guaranteed to always be a fixed size and may change dynamically.

The canvas element should mimic the screen size and page position of the video element, via the user’s CSS pixel size, however, this should not be forced, as it is controlled by the player.

The canvas element will be a WebGL canvas, and therefore it is not possible to use any of the HTML5 2D drawing context methods.

**lcevcDilConfig**

This is an object block containing settings with which V-Nova LCEVC is initialised. It is automatically configured with optimal settings.


# AndroidX (ExoPlayer)

This article introduces how to add LCEVC decoding and rendering into androidx/media project, Google's open source Android platform media libraries specifically ExoPlayer.

## Supported stream format

1. The solution utilizes`MediaCodec`for  decoding the base video stream so any codec supported by `MediaCodec` on the device can be supported for its LCEVC enhanced stream. Please consult [this page](https://developer.android.com/media/platform/supported-formats) to understand Android native codec support. For example, LCEVC enhanced H264 stream in local file (MPEG-TS `.ts`, MPEG-4 `.mp4`), as well as in HLS and DASH, are well supported. However, formats such as AV1 are only available on Android 10 and later, so playback of LCEVC-enhanced AV1 streams in this implementation faces the same limitation.
2. LCEVC data can be carried within the single video track using the SEI NAL unit type for MPEG codecs such as H.264, HEVC, and VVC. Alternatively, it can be contained within separate tracks, as seen in an `.mp4` video source where one track carries the base video stream and another track carries the LCEVC video stream. However, it's crucial to note that the current implementation only supports the use of a single video track carried within SEI data.

### Supported AndroidX (ExoPlayer) version

Since ExoPlayer version 2.13, the `MediaCodecAdapter`interface has been introduced and remained stable. The implementation of the LCEVC decoder extension relies on this interface, enabling seamless integration of the solution into media applications using ExoPlayer 2.13 and later versions.&#x20;

## Design overview

The provided class diagram offers a high-level depiction of how LCEVC decoding and rendering are integrated into the AndroidX framework:

1. LCEVC decoding and rendering is added into ExoPlayer as a decoder extension library, by use the LCEVC decoder native library.
2. `LCEVCSynchronousMediaCodecAdapter` is the entry point class, which inherits the interface `MediaCodecAdapter`defined in ExoPlayer core library. Its implementation is by use `MediaCodec` API to decode the base video stream, use LCEVC decoder native library to decode LCEVC video stream and render the final full resolution video frame.

<figure><img src="https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/xdDLkBTB19RxbndmK4jX/exoplayer-option-2-LcevcMediaCodecAdapter-more-detail.png" alt=""><figcaption><p>decoder_lcevc extension architecture</p></figcaption></figure>

## How to deploy

### Get decoder\_lcevc extension library

For media application already migrated to AndroidX project the integration can be quite straight forward:

1. Fetch specific version of Google AndroidX libraries in your project build.gradle i.e.

   ```
   def mediaVersion = "1.0.1"
   implementation "androidx.media3:media3-exoplayer:$mediaVersion"
   implementation "androidx.media3:media3-exoplayer-dash:$mediaVersion"
   ```
2. Fetch the matching version of V-Nova decoder\_lcevc library  by adding following into build.gradle i.e:

   ```
   def lcevcDecoderExtensionVersion = “1.0.1.hashcode”
   implementation "androidx.media3:media3-decoder-lcevc:$lcevcDecoderExtensionVersion "
   ```

### Code to enable LCEVC

Once you've followed the instructions above to fetch the AndroidX and decoder\_lcevc libraries, the next step is to tell ExoPlayer to use decoder\_lcevc by creating the `LcevcRenderersFactory` and create ExoPlayer instance:

```
renderersFactory = LcevcRenderersFactory(applicationContext)
player = ExoPlayer.Builder(this)
            .setRenderersFactory(renderersFactory)
            .build()
```


# DASH.js

LCEVC playback in DASH.js

LCEVC decoding and playback are available through **DASH.js**, the JavaScript-based reference player for MPEG-DASH content in web browsers.

By visiting the DASH.js sample playback site below, users can easily try LCEVC stream playback in DASH.js:\
<https://reference.dashif.org/dash.js/nightly/samples/index.html#MPEG5Part2LCEVC>

Online documentation and samples are also provided to demonstrate how MPEG-5 Part 2 LCEVC can be used for scalable delivery, where higher-resolution representations are generated by applying LCEVC data to corresponding lower-resolution base streams. The player allows switching between base and LCEVC representations:

<figure><img src="https://841975049-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FII3HaKNLFEqGvuN59zwF%2Fuploads%2F1T05RUC6zFVkgpWzrtje%2Flcevc%20in%20dash.png?alt=media&amp;token=8c0de6c0-7ace-4b53-870f-5c59f6c79a51" alt=""><figcaption></figcaption></figure>

<https://reference.dashif.org/dash.js/nightly/samples/lcevc/lcevc-dual-track.html>

The source code included in these samples provides practical guidance on how to enable and integrate LCEVC stream playback in DASH.js.


# Android open-source project

## **Introduction**

This page summarizes the steps for using MPEG-5 LCEVC with the Android Open-Source Project (AOSP). Android provides a well-defined framework for media applications to interact with the system at an operating system level, outlined in the diagram below. The Stagefright media playback engine incudes built-in software-based codecs for popular media formats at a native level. Stagefright audio and video playback features include integration with OpenMAX (OMX) codecs, session management, time-synchronized rendering, transport control and DRM. More information is available in the [Android Media framework documentation](https://source.android.com/devices/media).

V-Nova LCEVC integration fits at this OMX component level.

![Android media architecture ](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/i8Iv7tHsNAA9I4xYlPIl/0.png)

The approach for AOSP integration is via a plugin for OMXMaster named ***libstagefright\_lcevc\_plugin.so**,* which is loaded via OMXMaster’s API function *`OMXMaster::addPlugin`.*&#x20;

![Loading a new PlugIn in OMXmaster](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/81aLvxLSTVF89ZmclliJ/image.png)

The plugin is an implementation of `OMXPluginBase` that registers LCEVC specific components held in independent files. The current implementation provides support for enhancement of AVC/h.264 and HEVC encoding and decoding through the following components:

Decoder&#x20;

* `"OMX.vnova.video.decoder.avc-lcevc"`&#x20;
* `"OMX.vnova.video.decoder.hevc-lcevc"`

Encoder

* `"OMX.vnova.video.encoder.avc-lcevc"`
* `"OMX.vnova.video.encoder.hevc-lcevc"`

With this approach the changes to AOSP are minimal and confined to a standard way to loading new functionality. They can be applied to any fork of the AOSP project through a simple patch provided by V-Nova on its[ github repository](https://github.com/v-novaltd).  If you don't already have access to the repository please speak to your V-Nova representative or submit a request [here](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/).

This plugin allows the LCEVC components to be distributed and updated easily and separately by the provision of a simple .so which V-Nova provides together with the necessary scripts.

![LCEVC OMX plug-in and modification of OMXMaster](https://content.gitbook.com/content/II3HaKNLFEqGvuN59zwF/blobs/7GGv7hrC5ILOrravpVzP/image.png)

## **Summary**

Installing any AOSP packages involves replacing the operating system on the host devices. This should be done with care by experts as errors could render the device inoperable. The steps to do this are generally:

1. Download all packages
2. Prepare device and platform tools
3. Flash the LCEVC AOSP image onto the host
4. Install OMX components to host

In most cases, V-Nova will provide a patch for a customer's fork of the AOSP project. Fully built AOSP images are also available.

## AOSP image

### **Patch**

The best way to modify your AOSP fork is by applying the OMXMaster modification as a patch provided by V-Nova on its [github repository](https://github.com/v-novaltd/). The patch is non-invasive, minimal and uses the constructs made available by the Stagefright framework to load the necessary media functionality.

### Pre-packaged by V-Nova

V-Nova can provide a full image of ASOP already patched with the necessary modifications to load the necessary LCEVC plug-in. To date, V-Nova have added support for a few chipsets, such as the snapdragon-855, Google Pixel 4a (snapdragon-730G) and various emulators.&#x20;

The AOSP changes, software and steps to install LCEVC components are as agnostic as possible from the underlying chipsets to easily expand to other integrations.

### **Preparing devices and platform tools**

Some devices need more preparation than others, particularly if the host device is a mobile phone. Possible tasks include:

* Install Google platform tools (ADB and fastboot) on control PC
* Enable OEM unlocking and USB debugging on the host device
* Set up USB drivers on control PC (windows 10 users)
* Unlocking the bootloader
* Reverting to stock factory operating system

Specific instructions for devices are included in the documents within the packages.

### **Installing the AOSP image on the device**

Prerequisites

* LCEVC enabled AOSP image
* Terminal on control PC
* adb and Fastboot connections to the device

**Installing LCEVC enabled image**

* Enter the BootLoader (fastboot) with adb reboot bootloader
* In the terminal navigate to the location where you saved the ZIP file.
* Now enter fastboot -w update aosp\_\<chipset>-img-android-\<version>-lcevc-1.zip
* The device will reboot into the fastbootd prompt. Wait as the fastboot command is still running in the background, it will ask you to press a button to continue.
* The device will reboot itself and perform a fresh setup of Android, follow it through by Skipping as much as possible.
* Upon booting the device you'll see an LCEVC Enhanced logo.

## **LCEVC OMX Components**

Prerequisites

* Python 3.7
* adb connection to the device

Together with LCEVC OMX components for encoding and decoding, V-Nova provides all of the scripts necessary to install them, check them, uninstall them.

**Note**: in the paragraphs below we show the examples for a Snapdragon 855 chipset. Steps for other chipsets will be equivalent.

### **Installing the components**

Extract the build of LCEVC OMX components into some location

`tar -xzf lcevc-omx-snapdragon-855-<version>-<date>.tar.gz`

On Windows use 7Zip or equivalent. Within the package you will find the following directory structure:

`doc/`\
`emulator/`\
`scripts/`\
`snapdragon-855/`

including documentation, scripts and the actual OMX binaries.‌

Install the python requirements for installation script:

on Linux:

`python3 -m venv env`\
`source env/bin/activate`\
`pip install -r scripts/omx/requirements.txt`

on Windows:

`python3 -m venv env`\
`cd env/scripts/`\
`activate.bat`\
`cd ../../`\
`pip install -r scripts/omx/requirements.txt`

After making sure requirements are satisfied, installation of the OMX components is done simply using the [**omx.py**](http://omx.py/) script:‌

`python3 scripts/omx/omx.py install --host-path [--device <serial>]`

* `--host-path` is the path to the .tar.gz file with the OMX components
* `--device <serial>` - target device if multiple attached

For more information about the syntax of **omx.py** do:

`python3 scripts/omx/omx.py install --help`

### **Checking that components are installed**

Make sure the python requirements are installed.

After following the process above, check that the LCEVC OMX components are correctly installed on the system. You can do this using the scripts provided.

`python scripts/omx/omx.py info`

The output should look something like this (note that the actual checksum will change with releases of the components):

Verifying installation:

`> adb shell md5sum /vendor/lib/libstagefright_lcevc_plugin.so`\
`23ae53bd690917cc4fb7e3be5d223cc3 /vendor/lib/libstagefright_lcevc_plugin.so -> OK`\
`> adb shell md5sum /vendor/lib/libOMX.vnova.video.encoder.lcevc.so`\
`bd1f9931442a0362ea488c213117cd06 /vendor/lib/libOMX.vnova.video.encoder.lcevc.so -> OK`\
`> adb shell md5sum /vendor/lib/libOMX.vnova.video.decoder.lcevc.so`\
`b5a67d95cd1cccdde4d0a728ef447556 /vendor/lib/libOMX.vnova.video.decoder.lcevc.so -> OK`\
`> adb shell md5sum /vendor/etc/media_codecs.xml`\
`f4ced8075048178d497ae6b394910259 /vendor/etc/media_codecs.xml -> OK`\
`> adb shell md5sum /vendor/etc/media_codecs_lcevc_enc.xml`\
`d18c10022385177466af1a6dad1f1ac9 /vendor/etc/media_codecs_lcevc_enc.xml -> OK`\
`> adb shell md5sum /vendor/etc/media_codecs_lcevc_dec.xml`\
`600d19dabc11cda832b67bed6147163d /vendor/etc/media_codecs_lcevc_dec.xml -> OK`\
`> adb shell md5sum /vendor/bin/lcevc_omx_test`\
`2622ea3acfe9edc4f6e5fd1349a0e890 /vendor/bin/lcevc_omx_test -> OK`\
`> adb shell md5sum /vendor/lib/liblcevc_eil.so`\
`7940d978d5e544831e774fc0622ae202 /vendor/lib/liblcevc_eil.so -> OK`\
`> adb shell md5sum /vendor/lib/liblcevc_dil.so`\
`fd52f83eb88e8492d5d0c1509a3283a1 /vendor/lib/liblcevc_dil.so -> OK`\
`> adb shell md5sum /vendor/lib/liblcevc_epi.so`\
`eaeadd5312bbe5cf34507e415bbc8aed /vendor/lib/liblcevc_epi.so -> OK`\
`> adb shell md5sum /vendor/lib/liblcevc_dpi.so`\
`293ce36461a482a892409fc1c9f34817 /vendor/lib/liblcevc_dpi.so -> OK`

### Uninstalling components

Make sure the python requirements are installed.

Uninstall OMX components via [**omx.py**](http://omx.py/)

`python3 scripts/omx/omx.py uninstall [--device <serial>]`

* `--device <serial>` - target device if multiple attached

## Using the components

V-Nova provides several applications to test and verify encoding and decoding of LCEVC video using the OMX components installed. Currently, these applications are:

* **An Encoder** a simple encoding application that allows LCEVC enhancement to be set as well as calculating several useful statistics:

**lcevc\_erp\_android\_prod.apk**

* **A Decoder** a modified version of Android ExoPlayer that uses the LCEVC OMX components:

**Exo-OMX-r\<version>-OMX-\<date>-noExtensions-withPs-release.apk**

However, **any application** on the device can use the LCEVC OMX components through the Android MediaCodec API. The LCEVC OMX components are listed in the *`media_codecs_lcevc_enc.xml`* and *`media_codecs_lcevc_dec.xml`* files. They can be called explicitly by name using one of many APIs Android make available (e.g *`createByCodecName`*) or by providing roles, here are they names:

* `OMX.vnova.video.encoder.avc-lcevc`
* `OMX.vnova.video.encoder.hevc-lcevc`
* `OMX.vnova.video.decoder.avc-lcevc`
* `OMX.vnova.video.decoder.hevc-lcevc`

Roles according to their names:

* `video_encoder.avc`
* `video_encoder.hevc`
* `video_decoder.avc`
* `video_decoder.hevc`

Creating the module based on name is easy, just specify the name of the component to use and it will be created, this method is recommended. Creating the module by role can be different depending on what other components are installed on the operating system.

**Note:** It should also be noted that LCEVC modules should currently only be used when LCEVC-enhanced content is available, for example the LCEVC decoder will not be able to decode vanilla AVC/HEVC bitstreams, only LCEVC-enhanced bitstreams.


# GStreamer

As of September 2024, an initial integration of V-Nova's LCEVC Encoder and Decoder into the GStreamer framework is available. Please refer to the GStreamer 1.26 Release Notes: <https://gstreamer.freedesktop.org/releases/1.26/> :

* **LCEVC encoder and decoder elements** based on V-Nova's SDK libraries were added, including support in `h264parse` for **extracting the enhancement** **layer from H.264 and decoding it** via a `lcevch264decodebin` element.

This documentation provides a high level overview of the current integration and how to enable LCEVC encoding and decoding with GStreamer.&#x20;

### Integration Overview

* For LCEVC encoding with GStreamer user should download a trial pre-built V-Nova's LCEVC Encoder SDK and relevant base encoder plugin libraries directly just by signing up at: [https://download.v-nova.com](https://download.v-nova.com/). After that users should follow the  guide that can be find under `ext\lcevcencoder \README.md` under GStreamer download packager, to build the GStreamer LCEVC encoder plugin.
* For LCEVC decoding GStreamer has integrated V-Nova maintained open source project <https://github.com/v-novaltd/LCEVCdec>. User should follow the  guide that can be find under `ext\lcevcdecoder\README.md` under GStreamer download packager, to build the GStreamer LCEVC decode plugin.

Please note that in GStreamer 1.26 release LCEVC data is stored in only H264 base codec Supplemental Enhancement Information (SEI) NAL unit from both encoder and decoder point of view.&#x20;


# LCEVC encoder build

### Download and Install the LCEVC Encoder SDK

1. Register and download the SDK package:
   * Visit: <https://download.v-nova.com>
   * Select the appropriate pre-built `encoder_sdk.tar.gz` package (Ubuntu 20, Ubuntu 22, or Windows).
2. Extract and install SDK components:

   ```bash
   bashCopyEdit$ cd Downloads
   $ tar xvf encoder_sdk.tar.gz
   ```
3. Install the headers and libraries (example for Linux):

   ```bash
   bashCopyEdit$ cp -v include/*.h /usr/local/include
   $ cp -v *.so /usr/local/lib
   ```
4. Create a `lcevc_eil.pc` file for `pkg-config`:

   ```ini
   iniCopyEditprefix=/usr/local
   includedir=${prefix}/include
   libdir=${prefix}/lib

   Name: lcevc_eil
   Description: LCEVC Encoder EIL library
   Version: 3.11.3
   Libs: -L${libdir} -llcevc_eil
   Cflags: -I${includedir}
   ```
5. Install the `.pc` file:

   ```bash
   bashCopyEdit$ cp -v lcevc_eil.pc /usr/local/lib/pkgconfig
   ```

### Build the LCEVC GStreamer Plugin

To enable the LCEVC encoder plugin during GStreamer build:

1. Navigate to your GStreamer project directory:

   ```bash
   bashCopyEdit$ cd GStreamer
   ```
2. Configure the build with Meson:

   ```bash
   bashCopyEdit$ meson setup $BUILD_DIR --pkg-config-path=$INSTALL_DIR/lib/pkgconfig -Dgst-plugins-bad:lcevcencoder=enabled
   ```
3. Build using Ninja:

   ```bash
   bashCopyEdit$ ninja -C $BUILD_DIR
   ```

***

### Run the GStreamer LCEVC Encoder Pipeline

Test the LCEVC encoding with a simple pipeline:

```bash
bashCopyEdit$ gst-launch-1.0 videotestsrc pattern=ball num-buffers=1024 \
! video/x-raw,width=1920,height=1080,format=I420,framerate=25/1 \
! lcevch264enc plugin-name="x264" plugin-props="preset=superfast;tune=zerolatency" \
! h264parse ! mp4mux ! filesink location=lcevc-sample.mp4

```


# LCEVC decoder build

### Build and Install the LCEVC Decoder SDK

1. Clone the repository:

   ```bash
   bashCopyEdit$ git clone https://github.com/v-novaltd/LCEVCdec.git
   $ cd LCEVCdec
   ```
2. Build and install the SDK:

   ```bash
   bashCopyEdit$ mkdir $BUILD_DIR
   $ cd $BUILD_DIR
   $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR ..
   $ cmake --build .
   $ cmake --install .
   ```

***

### Build the LCEVC GStreamer Decoder Plugin

```bash
bashCopyEdit$ cd GStreamer
$ meson setup $BUILD_DIR --pkg-config-path=$INSTALL_DIR/lib/pkgconfig -Dgst-plugins-bad:lcevcdecoder=enabled
$ ninja -C $BUILD_DIR
```

***

### Run a GStreamer LCEVC Decoder Pipeline

Basic decoding pipeline:

```bash
bashCopyEdit$ gst-launch-1.0 filesrc location=/home/user/lcevc-sample.mp4 \
! qtdemux ! h264parse ! openh264dec ! lcevcdec ! videoconvert ! autovideosink
```

With automatic element discovery:

```bash
bashCopyEdit$ gst-launch-1.0 playbin uri=file:///home/user/lcevc-sample.mp4
```


# Getting started

In video streaming, *"carriage"* refers to the method by which encoded video data is packaged on the server side and transmitted for playback on the client side. This process involves structuring the video data into compatible formats, applying necessary protocols, and organizing metadata so that the client can decode and render the video smoothly.

This section covers LCEVC carriage solutions and examines their impact on components across end-to-end video systems. It explores how LCEVC integrates with existing streaming ecosystems, from encoding and packaging to playback, and addresses the impacts on each stage of the video delivery pipeline.

{% content-ref url="/pages/k3CjgehtYGN3NAVAOSWs" %}
[Overview of LCEVC carriages](/technologies/mpeg-5.lcevc/carriage/overview-of-lcevc-carriages)
{% endcontent-ref %}

{% content-ref url="/pages/UgOs2ARMqlNHtk5bYUVv" %}
[Carriage implementations](/technologies/mpeg-5.lcevc/carriage/carriage-implementations)
{% endcontent-ref %}


# Overview of LCEVC carriages

## Introduction <a href="#toc199947738" id="toc199947738"></a>

As an MPEG standard ([ISO/IEC 23094-2](https://www.iso.org/standard/79143.html)), LCEVC uses a NAL (Network Abstraction Layer) format for its encoded frame data, in the same way AVC, HEVC, EVC and VVC do. However each MPEG standard defines its own NAL format. Since LCEVC is an enhancement codec, its encoded data need to be carried as an addition to the base video it enhances. The LCEVC AU (Access Unit) consists of a single LCEVC NAL unit of type either LCEVC IDR or LCEVC non-IDR. This LCEVC NAL unit, which enhances a corresponding base AU, can be multiplexed with the base in two ways:

1. As a separate ES (Elementary Stream) track in the same container (e.g. Transport Stream ES PID, or track in ISOBMFF/MP4), or delivery system (HLS, DASH);
2. As some form of metadata in the same elementary stream as the base (e.g. using ITU-T T.35 metadata);

## Standard defined carriages <a href="#toc199947738" id="toc199947738"></a>

Carriage in MPEG-2 Transport Stream (TS) and in ISO Base Media File Format are defined in amendments that have been incorporated in the relative ISO standards (see below). These methods also partially cover delivery systems that are based on them, such as HLS (HTTP Live Streaming) and MPEG-DASH. In all these use cases the standard compliant carriages entail the use of an additional ES for LCEVC.

### MPEG-2 Transport Stream dual-PID <a href="#toc199947739" id="toc199947739"></a>

Support for carriage of LCEVC within MPEG-2 Transport Stream was introduced in [ISO/IEC 13818‑1:2022/Amd 1:2023](https://www.iso.org/standard/83253.html), and incorporated in the main standard starting from the 2023 edition ([ISO/IEC 13818-1:2023](https://www.iso.org/standard/87619.html), also as [ITU-T H.222](https://www.itu.int/rec/T-REC-H.222.0)).

What the standard mandates is the carriage of the LCEVC Annex B bitstream as separate PES (Packetised Elementary Stream) PID, identified with stream type 0x36 (see Table 2-34). In the general case of a Multi Program TS there can be multiple LCEVC PIDs associated with base ESes of different programs therefore, in order to associate each LCEVC ES with its base ES, two new extension descriptors have been added. Those descriptors provide high level metadata and a tag, as a mechanism to associate LCEVC and base ES.

LCEVC and base ESes signal tags by means of two new extension descriptors. Matching tags identify which LCEVC ES enhances which base ES.

Specifically, an *LCEVC\_video\_descriptor* is used in the LCEVC ES component loop in the PMT to specify the specific tag for that base and enhancement association (the *lcevc\_stream\_tag*). The base ES component will have an *LCEVC\_linkage\_descriptor* in its PMT loop, to specify one or more *lcevc\_stream\_tag* that signal the corresponding LCEVC ES needed to enhance this base bitstream.

<figure><picture><source srcset="/files/j6Q1FPDCSxYDe7YqMQKe" media="(prefers-color-scheme: dark)"><img src="https://841975049-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FII3HaKNLFEqGvuN59zwF%2Fuploads%2FIxqbpqOeqkeh166pYUib%2Flcevc_diagram_light.svg?alt=media&amp;token=c0aede3e-53ba-40aa-a104-bbdb7934d4d6" alt=""></picture><figcaption></figcaption></figure>

As a design choice, the coupling between the LCEVC and the base ES is not based on their PID values, in order for those associations to remain valid across remuxing operations. This is because upon remuxing PIDs may change, while descriptor contents are generally passed through, which keeps the stream tag matches valid.

At the sample level, matching between LCEVC and base AUs is performed by means of the Presentation Time Stamp (PTS).

### ISO Base Media File Format dual-track

Carriage of LCEVC in ISO Base Media File Format (ISOBMFF) was introduced in [ISO/IEC 14496-15:2022/Amd 1:2023](https://www.iso.org/standard/83337.html), incorporated in the main standard starting from the 2024 edition ([ISO/IEC 14496-15:2024](https://www.iso.org/standard/89118.html)).

Again, the LCEVC sample stream is carried in a separate track, for which a new sample entry has been defined (see Section 13). Similarly to the other MPEG standards an LCEVC DecoderConfigurationRecord has been defined (see Section 13.3.3.2), which conveys general metadata and a loop of NAL unit arrays to carry LCEVC config blocks (global config, sequence vconfig and additional info).

Here the coupling between base and enhancement (LCEVC) tracks is achieved by means of the `sbas` (scalable base) atom, as defined in ISO/IEC 14496-12 and Subclause 6.5.1 of the LCEVC Amendment. The sbas atom, to be placed in the LCEVC track loop, signals the track Id of the corresponding base track it enhances.

At the sample level, matching between LCEVC and base AUs is performed by means of the composition time.

#### CMAF profile dual-track

A CMAF profile for LCEVC has been introduced in [ISO/IEC 23000-19:2024/Amd 1:2024](https://www.iso.org/standard/85642.html), published in July 2024.

The LCEVC profile is built upon (i) the “dual-track” structure as described in Section 2.1.2 and (ii) the existing CMAF construct of Dependent CMAF Tracks, as defined in Clause H.1 of [ISO/IEC 23000-19:2024](https://www.iso.org/standard/85623.html).

### **Non-Scalable MPEG-DASH**&#x20;

In this approach, the base-layer video (AVC, HEVC, or VVC) and the LCEVC enhancement layer are carried together in a single media segment as dual-track within the ISO-BMFF or CMAF profile described above. In the DASH manifest, a single DASH Representation is used, which references this dual-track media segment.

The DASH manifest structure largely mirrors that of a single-layer codec. Please see the descriptions below for clarification of key attributes:

* @codecs = \<base layer codec syntax>
* @width/height = \<Base layer video width/height>
* @bandwidth = \<combined bit rate of the LCEVC + base representation>

A sample manifest file is available: <https://d3mfda3gpj3dw1.cloudfront.net/Test_Streams/3.14/LCEVC_H264/BBB_DualTrack_DASH_StaticResiduals/master.mpd>

### Scalable MPEG-DASH (@dependencyId)

*Note: This solution is pending incorporation in the next DASH-IF IOP (Interoperability Points) v5.1, see "Features and Additions on top of Latest Version" in* [*https://dashif.org/guidelines/iop-v5/*](https://dashif.org/guidelines/iop-v5/) *at approved amendment:* [*DASH-IF Feature: LCEVC in DASH (published 17 Dec 2025)*](https://github.com/user-attachments/files/24211534/SVTA-DASH-IF-Feature-LCEVC-1.0.0.pdf)*. The scalable LCEVC feature uses tools from the existing* [*ISO/IEC 23009-1*](https://www.iso.org/standard/83314.html) *DASH specification. No modifications of the DASH norm have been proposed to support this LCEVC scalable DASH solution.*

This approach is "scalable" in allowing the client to either load the base only and play the base video or load both the base and the LCEVC enhancement to play enhanced, higher quality, video. Put differently, the client can also play base video only without having to load and discard additional data for the LCEVC enhancement, as would be the case in the non-scalable approach described above. This scalable feature is achieved without introducing new syntax elements in the DASH specification, making it compatible, at minimum for base only, with virtually any DASH client.

To offer scalability base and LCEVC encoded media must be delivered separately, with LCEVC only Representations and base Representations that respectively form separate Adaptation Sets. In the manifest the @dependecyId attribute is added to each LCEVC Representation to signal which base Representation it enhances. The @dependencyId aware DASH client will have to load the dependecyId signalled Representation (defined "Complementary" Representation in the DASH standard), which is the scalable base, in addition to the parent one, i.e. the LCEVC Representation.

The supplemental property “adaptation-set-switching:2016” is added to both video and LCEVC Adaptation Sets, with each one pointing at the other, to signal the client that the switching set shall be the union of all the representations of both Adaptation Sets. Clients not supporting this supplemental property, and/or not supporting LCEVC as base codec (codecs string `lvc1`), would only switch among the Representations found in the base Adaptation Set.

With focus on the LCEVC Adaptation Set, following are the key attributes at Adaptation Set and Representation level:

* @contentType = ‘video’
* @mimeType = ‘video/mp4’
* @codecs = ‘lvc1’
* @frameRate = <>
* @dependencyId = <>
* @width/height = \<LCEVC enhanced video width/height>
* @bandwidth = \<combined bit rate of the LCEVC + base representation>
* @sar = \<sample aspect ratio as signalled in LCEVC Video Usability Information (VUI)>

For all the details on the LCEVC Scalable DASH solution please refer to the DASH IOP approved amendment [*DASH-IF Feature: LCEVC in DASH (published 17 Dec 2025)*](https://github.com/user-attachments/files/24211534/SVTA-DASH-IF-Feature-LCEVC-1.0.0.pdf).

***

## Proprietary solutions <a href="#toc199947742" id="toc199947742"></a>

The carriage cases described in this section are based on proprietary solutions, that either embed LCEVC NAL units in SEI messages or interleave them with NAL units of other codecs.

### ITU-T T.35 user-registered SEI

The LCEVC AUs can be embedded within an SEI (Supplemental Enhancement Information) message of other MPEG ES using the ITU-T T.35 user data registered syntax already specified in MPEG codecs. Specifically, the SEI  NAL Unit Type are:

* **H.264 (AVC):** NAL unit type **6,** defined in ISO/IEC 14496-10;&#x20;
* **H.265 (HEVC):** Prefix SEI – NAL unit type **39,** defined in ISO/IEC 23008-2;
* **H.266 (VVC):** Prefix SEI – NAL unit type **23,** defined in ISO/IEC 23090-3.

V-Nova got assigned a manufacturer code under the UK Register of Manufacturer Codes for H.221/H.245/T.120, which can be used for this purpose.

With that code, the payload has the following format.

```
sei_payload( 0x04, payloadSize ) = user_data_registered_itu_t_t35 ( payloadSize )
```

Note that the itu\_t\_t35\_country\_code is followed by three additional fields specific to the National Register for the UK.

*Table – ITU-T T.35 syntax with the values specified by the V-Nova manufacturer code registered in the UK*

<table><thead><tr><th width="464.5" valign="top">Syntax</th><th width="147" valign="top">Descriptor</th><th valign="top">Value</th></tr></thead><tbody><tr><td valign="top">user_data_registered_itu_t_t35 ( payloadSize ) {</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">    itu_t_t35_country_code</td><td valign="top">b(8)</td><td valign="top">0xB4</td></tr><tr><td valign="top">    itu_t_t35_payload_header {</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">         t35_uk_country_code_second_octet</td><td valign="top">b(8)</td><td valign="top">0x00</td></tr><tr><td valign="top">         t35_uk_manufacturer_code_first_octet</td><td valign="top">b(8)</td><td valign="top">0x50</td></tr><tr><td valign="top">         t35_uk_manufacturer_code_second_octet</td><td valign="top">b(8)</td><td valign="top">0x00</td></tr><tr><td valign="top">    }</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">    i = 4</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">    do {</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">         itu_t_t35_payload_byte</td><td valign="top">b(8)</td><td valign="top"> </td></tr><tr><td valign="top">         i++</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">    } while ( i &#x3C; payloadSize )</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">}</td><td valign="top"> </td><td valign="top"> </td></tr></tbody></table>

The LCEVC AU is the payload of the above SEI message.

### ITU-T T.35 OBU

In a similar way to MPEG standards, a LCEVC AU can be embedded as metadata in an AV1 OBU stream (see [AV1 Bitstream & Decoding Process Specification](https://aomediacodec.github.io/av1-spec/av1-spec.pdf)). The OBU syntax comprises a header and a payload. The header is as follows:

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top">obu_header( ) {</td><td valign="top">Descriptor</td><td valign="top">Value</td></tr><tr><td valign="top">      obu_forbidden_bit</td><td valign="top">f(1)</td><td valign="top">0</td></tr><tr><td valign="top">      obu_type</td><td valign="top">f(4)</td><td valign="top"> </td></tr><tr><td valign="top">      obu_extension_flag</td><td valign="top">f(1)</td><td valign="top"> </td></tr><tr><td valign="top">      obu_has_size_field</td><td valign="top">f(1)</td><td valign="top"> </td></tr><tr><td valign="top">      obu_reserved_1bit</td><td valign="top">f(1)</td><td valign="top">0</td></tr><tr><td valign="top">      if ( obu_extension_flag == 1 )</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">            obu_extension_header( )</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">}</td><td valign="top"> </td><td valign="top"> </td></tr></tbody></table>

For the carriage of LCEVC the obu must be of metadata type, which is identified by obu\_type equal to 5. Among the types of metadata OBUs defined in AV1 there is an ITU-T T.35 type that is to be used for LCEVC carriage in a similar way as for the MPEG encapsulation described above.

In the LCEVC case the OBU payload is as follows:

<table data-header-hidden><thead><tr><th width="344.3333740234375" valign="top"></th><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top">metadata_obu( ) {</td><td valign="top">Descriptor</td><td valign="top">Value</td></tr><tr><td valign="top">      metadata_type   /* equal to METADATA_TYPE_ITUT_T35 */</td><td valign="top">leb128()</td><td valign="top">0x04</td></tr><tr><td valign="top">      if ( metadata_type == METADATA_TYPE_ITUT_T35 )</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">            metadata_itut_t35( )</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">}</td><td valign="top"> </td><td valign="top"> </td></tr></tbody></table>

with the payload as:

<table data-header-hidden><thead><tr><th width="347" valign="top"></th><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top">metadata_itut_t35( ) {</td><td valign="top">Descriptor</td><td valign="top">Value</td></tr><tr><td valign="top">      itu_t_t35_country_code</td><td valign="top">f(8)</td><td valign="top">0xB4</td></tr><tr><td valign="top">      itu_t_t35_payload_uk_lcevc()</td><td valign="top"> </td><td valign="top"> </td></tr><tr><td valign="top">}</td><td valign="top"> </td><td valign="top"> </td></tr></tbody></table>

where the payload itu\_t\_t35\_payload\_uk\_lcevc() is the itu\_t\_t35\_payload\_header() and itu\_t\_t35\_payload\_byte(s) as in user\_data\_registered\_itu\_t\_t35 defined above for SEI.


# Carriage implementations

In this section, we provide a brief description of the carriages’ implementation with a focus on the standards-based solutions, in a few well-known player applications.

## AndroidX Media3 <a href="#toc199947745" id="toc199947745"></a>

V-Nova maintains a public fork of the AndroidX Media3 media player SDK (formerly known as ExoPlayer) in which the default decoder (MediaCodecAdapter) can be replaced by a wrapper (LcevcMediaCodecAdapter) that has the additional functionality to detect and decode LCEVC-encoded media. This fork is available here:

{% embed url="<https://github.com/v-novaltd/lcevc-androidx-media>" %}

In this fork, functionality is also added to support the dual-track delivery mechanism for base and LCEVC enhancement, as described in Section 3.1.2 above. The relative code changes are shown in this commit:

<https://github.com/v-novaltd/lcevc-androidx-media/commit/762cf0b4f4c73e9404c36f1972a54a408ffc7d58>

### MIME type support <a href="#toc199947746" id="toc199947746"></a>

In the current AndroidX Media3 integration, logic is implemented to detect when a stream is LCEVC enabled through the `lvc1` FourCC code for the dual-track use case. Not yet implemented is the functionality to subsequently open the appropriate `MediaCodec` using the relevant MIME type (e.g., `video/lcevc-avc`, `video/lcevc-hevc`, `video/lcevc-vvc`).

Going forward, the integration will be updated in the following way:

1. For an LCEVC enabled stream, where the player detects the stream as e.g. `lcevc + hevc`, it will query MediaCodec for `video/lcevc-hevc` MIME type support.
2. If MediaCodec reports *Yes*, then open MediaCodec with `video/lcevc-hevc`. This will allow integrators to set up an LCEVC decoder accordingly.
3. If MediaCodec reports *No*, then open MediaCodec with `video/hevc` as it is today. This will allow backward compatibility across devices that are not yet LCEVC-enabled.

This logic will be added to the AndroidX Media3 integration in the coming weeks.

### MP4 dual track support <a href="#toc199947746" id="toc199947746"></a>

Media decoding in Android is provided by the MediaCodec class, which interfaces encoding and decoding services implemented in native code. MediaCodec is a one-input-one-output block, therefore, input from two different streams is not supported out of the box. To circumvent this limitation a “joiner” solution has been used. The joiner prepends each sample of the LCEVC Enhancement track to the corresponding sample of the Base track, thus resulting in a single flow of samples, each new sample being the LCEVC sample followed by the base sample, to the reader and ultimately to MediaCodec. The design of the LCEVC NAL units allows a legacy reader (e.g., the MediaCodec implementation of a base decoder) to interpret them as “reserved” or “unspecified” and thus discard them, which results in the joiner not hindering the decoding of the base component.

The above-linked commit reveals all the code changes. In principle, the joiner solution can be described briefly as:

* Modifying the SampleQueue class to:
  * Add an `enhancedSampleQueue`, default null, as a link to a parallel queue for the samples from the enhancement (LCEVC) track;
  * Add a method `attachEnhancement()` to allow setting the `enhancedSampleQueue`;
  * Add a method `isEnhancement()` to allow a client to know if the queue is of a base or an enhancement;
  * Prepend the data read from the linked enhancement queue, if present, to the data read from the queue in the `read()` method, effectively performing the join operation;
* Adding logic to recognise the presence of the LCEVC track, implementing the signalling as defined in ISO/IEC 14496-15:2024;
  * LcevcConfig class
  * Parsable Atoms
  * Sbas atom and setting of scalableId (as is of the base track)
* Adding logic to process main and enhanced tracks in MP4, fragmented MP4, HLS segments, and DASH chunks;
* Adding logic to attach the enhancement `SampleQueue` to the base `SampleQueue`, based on the presence of tref/sbas signalling in MP4.


# Overview

### Introduction

VC-6 (SMPTE ST 2117-1:2020) is a flexible, multi-resolution, intra-only codec that employs data agnostic techniques for compression and can provide both lossless and lossy modes.

There are two significant differences between VC-6’s hierarchical coding techniques and those of discrete wavelet transform-based formats:

* VC-6 can use more elements in its decomposition scheme than in the actual picture. It is unnecessary to send all decomposition elements explicitly; instead, residual data is sent to correct already-calculated predictions. Residual data takes up fewer bits when encoded, setting this standard apart from wavelet transforms.
* VC-6 uses up-sampling techniques that might not be suitable for use in wavelet transforms.

VC-6 also employs additional hierarchical coding techniques that are distinct from those used in predictive coding, sub-band coding-based codecs and discrete wavelet transform-based formats. These techniques involve data structures called s-trees.

### Key attributes

#### Bitstream

* Support for up to 255 component data planes, e.g., colors and alpha, represented in individual component planes, each with any resolution smaller than 65,536 by 65,536. Component plane sizes are independent of each other.
* Prioritization of particular locations in a picture is possible.
* Simple, repetitive bitstream payload structure with a shared header and limited co-dependency.
* Extensive support for injection of metadata.

#### Encoder

* Potential for different encoder implementations to employ different methods to produce bitstreams that conform to this standard, as this standard only specifies a bitstream format and generic decoding processes.
* Ability of a conformant bitstream to specify any valid combinations of reduced and extended syntax options, so as to keep the choice of optimal bitstream format entirely an encoder’s decision.

#### Decoder

* Ability of decoders to independently select a resolution according to processor and display capability.
* Elimination of blocking artifacts.
* Power consumption/quality trade-off at decoder end.

#### Codec

* Lossless quality, assuming a sufficient bitrate.
* Nonlinear upsampling, configurable in both encoder and decoder.
* Efficient, multi-resolution format.

### VC-6 standard

The VC-6 codec standard is published as SMPTE ST 2117-1:2020.

### VC-6 tools and licensing

V-Nova provides a complete software development kit (SDK) for VC-6, which includes: software libraries for encoding and decoding VC-6 bitstreams, example applications, and an FFMPEG and Python integration. There are two versions of the SDK that support Linux and Windows, using either the CPU (x86) or GPU implementation.


# Codec description

VC-6 is fundamentally different to most industry mezzanine codecs because it’s based on repeatable s-tree structures rather than block-based DCT or Wavelet transforms. Also, it is not limited to three data planes, i.e., RGB or YUV, but can handle up to 255 separate data planes which can each be a different size or resolution. Each data plane is compressed separately by creating a hierarchy of different resolutions of an image and then encoding the residual differences between them with a uniquely low-complexity approach. These simple tree structures provide intrinsic capabilities that are well suited to modern computing techniques like massive parallelism.

![VC-6 RGB Planes example](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/xgxW9umAwbzusI1skXmx/0)

![](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/HzIiPBnA1BSzTA4bfBKr/1)

#### VC-6 encoding and decoding processes

Each plane of each source image (I0) is down sampled by ½ in height and width a number of times to produce lower resolution versions (I-1, I-2, I-3…) giving us what is referred to in the SMPTE standard as the plane stack, and each resolution is called an echelon. The number of echelons generated is configurable and is one way the balance between compression quality and computational complexity can be controlled.

The reverse process is carried out, starting with the lowest echelon, (let us assume for example it is -3), so I-3 is up sampled to generate B-2, a reconstructed version of I-2. Then B-2 is up sampled to give B-1, and B-1 is up sampled to give B0.

The differences between the source echelons and the reconstructed echelons, i.e. the errors introduced by the down sampling and up sampling processes are calculated for each echelon apart from the lowest one which has no reconstructed version. These are referred to as the residuals (R0, R-1, R-2….).

The lowest echelon and the residuals from the higher echelons are then encoded giving E-3, E-2, E-1 and E0. It is this encode process that provides the data compression.

![VC-6 Encoding](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/ilFtv0m879EpFJQTUe4G/2)

Unsurprisingly, decoding is a very similar process but in reverse. The VC-6 bitstream contains a compressed version of the lowest resolution echelon, and compressed sets of residuals, or the errors that are introduced when the base resolution image is successively up-sampled.

If a lower resolution version is all that is required, then it is not necessary to decode and process any of the higher echelons, making the decoding process quicker and needing less resources such as memory.

The downsampler is not defined by the VC-6 standard, which allows the VC-6 encoder designer to trade-off compression performance against computational efficiency and speed.

The choice of upsampler is however constrained by the standard and is signalled in the VC-6 bitstream. Currently the VC-6 standard defines 4 possible upsamplers which are:

* Nearest neighbor
* Bicubic
* Sharp
* Standard Nonlinear 9-Transformer

![VC-6 Decoding](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/lJynlQvpUA4CkxTo2loA/3)

#### VC-6 data encoding details

It is the encode stage that provides the data compression, and consists of the following processes:

**Directional decomposition -> Quantization -> S-Tree and Entropy Coding-> Range Encoding**

**Directional Decomposition**

The lowest resolution image and the residual images for the higher layers transformed into four directional versions, using directional transforms to generate A (average), H (horizontal), V (vertical) and D (diagonal) versions or echelons that are ½ the resolution. This is done to both produce data structures with greater probability of sequences of zeros, which can be efficiently encoded, and to allow any necessary quantisation of the data to be targeted to minimise the impact in terms of visually perceptible video compression artifacts.

![Directional decomposition](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/stqwOUJytw5zVDSuYSGc/4)

**Quantisation**

VC-6 supports lossless compression, which if selected means that there is no quantisation of picture data.

However for lossy compression a rate control function determines which elements to quantise and to what degree to achieve the required data compression whilst minimising the impact on the perceived quality of the image. To help achieve this, each directional echelon is divided into 16 x 16 tiles and based on the characteristics of these tiles they are put into groups or tile sets. Each tile set has the same level of quantisation applied to it.

The number of tile sets is not defined but is left for the VC-6 encoder to determine the optimum number considering that there is a cost associated with each tileset as the quantization parameters must be transmitted to the decoder in the VC-6 bitstream.

Grouping tiles of similar characteristics and applying the same degree of quantisation to them can help maintain a consistent appearance to identical or similar surfaces in the image.

**S-Tree representation and entropy encoding**

Each quantised 16 x 16 tile is represented by an S-tree structure. The diagram below illustrates how elements in the 16 x 16 grid are grouped in sets of fours and formed into a tree structure. This tree structure is a very efficient way of representing the 16 x 16 elements if the 16 x 16 array is sparsely populated i.e., a significant number of the elements are zero when entropy coding is applied.

![](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/zwIfCzwOeY7QnvRjWxTd/5)

![S-Tree structure for 16 x 16 grid](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/aPoPP7eKZbMPJS3h7GUN/6)

It is perhaps easier to describe how the entropy encoding of a sparsely populated array works by considering a 1D array as illustrated in the diagram below. This shows a sparsely populated 50x1 array with non-zero values in elements 9, 10, 11, 12, 20, 22, 23 and 24.

If the size of the array is known in advance, then the number of layers in the tree structure is implied, in this case 5, it is also assumed that any element that is not explicitly decoded has the vale 0.

The tree structure is described using metadata or T codes where:

* 1 means jump to the first node in the layer above
* 0 means continue to the next node in the current layer unless no further nodes so backtrack to the
* X means don’t care treat as zero

The process of decoding such a tree is called de-sparsification.

In this example the last three nodes in layer -3 are inactive as the array is known to have only 50 elements, and the tree structure can represent 256 elements, hence they are represented by x in the Tcode.

A similar approach is adopted by VC-6 to efficiently represent the 2D 16 x 16 tiles.

![Fifty element 1D array example](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/2ulHrAxZyJOR1OC2KpWP/7)

**Range encoding**

The final stage of compression involves the application of standard data compression techniques such as range encoding to produce the final VC-6 compressed bitstream.

#### VC-6 bitstream

The VC-6 bitstream consists of the following major parts:

* Primary Header
  * Fixed size of 168 bits giving the high-level stream information such as the format version, size, number of planes etc.
* Secondary Header
  * Variable size which is determined by some of the data values within the primary header and includes the heights and widths of each plane, and the number of echelons.
* Tertiary Header
  * Variable size which is determined by some of the data values within the primary and secondary headers.
* Payload

The VC-6 bitstream can be organised in several different ways depending upon the application. Figure 8 for example shows how it can be ordered to allow decoding of the lower resolution versions before the entire image has been received by the decoder in a streaming type of application. However if fast access to one particular image plane is important, then the payload can be ordered by plane rather than echelon (or resolution).

![VC-6 bitstream structure](https://content.gitbook.com/content/g9PKlUvkbpCzUNAxkswo/blobs/aU6uoXigm3OmB9QqZx1N/8)


# SDK

### Introduction

V-Nova’s VC-6 SDK includes a VC-6 codec library implemented in C++ and comes with a C-Style API. It provides encode and decode functionality as well as some additional utilities. There are two variants of the SDK for Linux and Windows: CPU (C++) and GPU (OpenCL, Metal).

* Linux OpenCL version
* Linux C++ version
* Windows OpenCL version
* Windows C++ versionclinfo

When GPU acceleration is not available, C++ provides an implementation that has been optimised to exploit the parallel processing capabilities of the latest CPU architectures.

### SDK content

The SDK is made up of the following components:

* Header Files
* Libraries
* Sample files
* Sample binaries and scripts
* API Reference
* Documentation

### Getting started (OpenCL)

To integrate and test VC-6 on a platform with GPU (OpenCL) acceleration, the user should ensure that at least one OpenCL-enabled device is set up on the development machine. A command-line tool named **clinfo** can be used to check up and running OpenCL devices.

### Clinfo

For Windows machines, binary releases can be downloaded from this [github repo](https://github.com/Oblomov/clinfo).

For Linux systems, *clinfo* can be installed via the package manager. For example on Ubuntu:

```
<apt-get install clinfo>
```

By running the **clinfo** command, you will either get a single line indicating you have 0 platforms in your system, or extensive details of each available platform and device. What is important are the number of platforms and devices in each platform, and if the desired device is listed, e.g., a dedicated Nvidia GPU. If the expected platform cannot be found, the device drivers should be installed.

### Driver installation

For dedicated GPUs, follow the target vendor's instructions. For an Intel CPU with Integrated GPU:

First check if the integrated GPU can be detected by clinfo, as the installation of OpenCL CPU drivers can override the Intel GPU drivers. We recommend using the integrated GPU than the CPU itself. Intel integrated GPUs 4000 series and above have the capability of running OpenCL. If an integrated GPU is not available and there are no OpenCL devices, install the appropriate driver from [this link](https://software.intel.com/content/www/us/en/develop/articles/opencl-drivers.html):

#### Windows

In [Intel Download Center](https://downloadcenter.intel.com/), select Processors or Graphics. Then select your operating system and processor/graphics model and finally download and install the driver. A system restart is probably required.

#### Linux

The user will be guided to a [GitHub repo](https://github.com/intel/compute-runtime) in which installing or building guides are provided.

### Sample code dry run

A simple program named *info\_init.cpp* can be found in the samples directory. If the development environment and target OpenCL runtime are set up correctly, you should be able to compile and run the program and get a console output with VC-6 SDK version and the OpenCL device picked for runtime with the following format:

**Codec version: \<ReleaseNumber-CommitHash >\<CodecType>\<OpenCL Device>**

### Selecting device

If multiple OpenCL-enabled devices are available on a machine, it is very likely that the output of the previous command does not show the intended device. This can be controlled by setting *OCL\_DEVICE* environment variable.

This environment variable can be used to specify the OpenCL device picked by the codec if more than one exists. The variable can be assigned both numbers and strings. If a number (starting from 0) *N* is used, the codec will pick the *(N-1)*&#x74;h device appearing in the *clinfo* output. A case-sensitive string can be also used and the codec will try to match that with device descriptions and pick the first match. For example, if an intel Core i9 CPU is the second listed device in *clinfo*, all three following settings will pick that device:

OCL\_DEVICE=1 *OR* OCL\_DEVICE=Intel *OR* OCL\_DEVICE=Core

### Kernel caching

Compiling OpenCL binaries for each device/codec-version combination will take a noticeable amount of time. This can be avoided by caching the compiled binaries in a temporary folder. The user can set this location by setting their desired address in *OCL\_BIN\_LOC* environment variable. The codec will both look for the prebuilt binaries and write them in this location if they are not already there.

The *info\_init* program can be used with *--build-kernels* option to compile and cache the binaries to *OCL\_BIN\_LOC* location. In this program it is also demonstrated how to use encrypted kernels.

In order to use the *info\_init* to create binaries from the encrypted kernels, first you should set the *OCL\_BIN\_LOC* variable to a writable directory, and then invoke the *info\_init* program with ***--build-kernels*** and set ***--kernel-dir-enc*** option to path to the encrypted kernels' directory.

### VC-6 codec concepts

At a high-level the VC-6 codec requires three components to operate:

* **Context**: an environment that contains a set of channels, streams and common objects (including the device, platform, context, etc).
* **Stream(s)**: encoder or decoder instances
* **Channel(s)**: interfaces to the streams

Auxiliary objects are required to setup these main objects and manage the details of the encoding process. The most important of these objects can be enumerated as:

* **Stream configuration**: contains the necessary information to setup a stream
* **Channel configuration**: contains the necessary information to setup a channel
* **Picture description**: holds dimensions and format data of a picture/video
* **VC-6 error type**: contains error code and other error information
* **Video reader**: an optional tool to read raw video (yuv or rgb) or VC-6 video frames. (Included in the utilities).

A basic VC-6 encoder program would create a context and a number of channels (per image/video). The streams, as codec instances, can be created explicitly and then become connected to channels via a process called *linking*. The developer can choose to let the codec handle the streams and their connections to channels for simple usecases. And finally, the encoding can take place by calling a set of channel-specific API calls. A high-level workflow of the program can be described as:

1. Creating a context
2. Creating channel configurations and video readers
3. Creating channels
4. Creating stream configurations and streams(optional)
5. Linking channels and streams
6. Performing the following loop:
   1. Begin channel input
   2. Preparing the input (preferably using video reader)
   3. End channel input
   4. Begin channel output
   5. Flush the channel
   6. Wait/Query channel output
   7. Discard and release the output

###


# Introduction

Welcome, this is a top level description of  V-Nova PresenZ.

#### W**hat is V-Nova PresenZ**

V-Nova PresenZ is a unique immersive movie and image format, allowing for 6 degrees of freedom into pre-computed images. It uses a revolutionary approach to 3D creation and rendering, enabling truly movie-quality immersive worlds with higher detail, better immersion, and more fluid, natural movement. By leveraging V-Nova’s advanced data compression technology, V-Nova PresenZ makes these rich, high-quality experiences practical to deliver, without compromising visual fidelity.
\
This allows content creators to craft immersive experiences like never before, while also enabling already rendered media to be re-imagined for VR. For studios and rights holders, this opens up an additional revenue stream, creating immersive experiences with minimal overhead.

#### Legacy documentation

We are currently updating all V-Nova PresenZ documentation. Some materials may not yet be live, so we have provided a link to the legacy documentation below. If you cannot find what you are looking for in the current resources, please refer to the legacy documentation, which may contain the information you need.

Legacy documentation: [here](https://support.presenzvr.com/docs/html/index.html#)

#### Error reporting

If you encounter any technical issues, our technical artists are available to assist. Please use the link below to contact support. We also welcome feedback and suggestions, which we review as part of our ongoing development.

Email us: <presenz-support@v-nova.com>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FKZd3VB10O95nnIh1iFpF%2FVN_presenZ.jpeg?alt=media&amp;token=dc79b902-e611-49fb-b5f2-e3ccc7f1abbb" alt=""><figcaption></figcaption></figure>


# Zone Of View

The area within which V-Nova PresenZ allows for head movement

**The Zone Of View (ZOV)**

Understanding the concept of the Zone of View (ZOV) is crucial to working with V-Nova PresenZ. In a standard precomputed rendering, a perspective camera projection is used, which is centered around a point of view (POV), or focal point. However, this type of rendering can cause issues in VR because the user’s perspective is locked to the POV, meaning that the user can’t see any objects that are not visible from this point of view. But in VR, the users can make head movements; those movements, like in real life, should change the perspective on the scene.  In VR, if we are locked to a single POV, this causes cybersickness, discomfort, and poor immersion.

V-Nova PresenZ, on the other hand, renders from a Zone Of View (ZOV) that gathers all the information that is visible from within this zone. The ZOV is a volume box with a default size of 1 by 1 meters and a height of 0.5 meters. However, this size can be adjusted.

The benefit of using the ZOV is that the user running the PresenZ Player can move freely inside this viewing volume and experience a proper perspective change on the pre-rendered 3D scene. This means that the user can explore the virtual world from different angles and viewpoints, like in real life, resulting in a more immersive and engaging VR experience.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FXj1wL3y0kOt6n4bAeX3y%2F1.jpeg?alt=media)

{% hint style="info" %}
***Note:** When the user is outside of the ZOV, it appears as if he was looking through a portal to the VR  scene.*
{% endhint %}


# Two-phase rendering

### **The two-phase system**

V-Nova PresenZ generates images through a two-phase system consisting of a detection phase and a render phase. Once these two phases are complete, we get one or several “.PrzRender” files. \
Then PresenZ Merger, also called the Prz\_Maker, combines the results to create a “.PRZ” file that can be viewed in VR by the PresenZ player. \
This “.prz” file is a compressed file, think of this like a “.jpeg” file, while a “.PrzRender” file is more like an “.exr” with all the aov and colour data present for compositing.

#### &#x20;1. **Detection phase**

During the detection phase, V-Nova PresenZ gathers all the necessary information to cover whatever is visible from the ZOV, optimizing this information to obtain the proper detail level as seen from the ZOV. This phase probes the 3D scene to define what render origins and directions have to be performed during the next phase (the render phase). At the end of the detection phase, V-Nova PresenZ generates a “.przDetect” file that is stored in the przDetect subfolder created automatically at your output render pass.

{% hint style="info" %}
*Note: Depending on the render engine, there might be multiple detect passes, for example, one for transparent and one for opaque objects.*
{% endhint %}

#### &#x20;**2. Render phase**

In the render phase, V-Nova PresenZ shoots sampling rays according to the origin and direction given by the detection phase. This phase applies lighting and shading, resulting in colour data and other AOVs (Arbitrary Output Variables) computations for the final output. The render phase generates a “.przRender” file that is stored in the przRender subfolder created automatically at your output render pass. The “.przRender” file is a RAW PresenZ image file that contains all the colour and AOV information in half float precision. This file can be manipulated for colour calibration and compositing. It supports  V-Ray colour correction for V-Ray users, but also other colour grading techniques like  [OCIO (OpenColorIO)](https://support.presenzvr.com/docs/html/Utilities/OCIOSupport.html#ociosupport), for instance.

{% hint style="info" %}
*Note: All compositing, tweaking, and post-processing will be done on the .prZRender files that ultimately will become your "master" copy of the movie before compression.*
{% endhint %}

**Prz\_Maker**

The PresenZ Merger/ Prz\_Maker converts the previously created “.przRender” file(s) to a “.PRZ” image file that can be viewed in the PresenZ player. The .przRender files carry uncompressed pixel data, which is compressed during the merging process, resulting in the .prz file in a smaller file size that can be easily handled by the player.

In summary, V-Nova PresenZ uses a two-phase system to generate images for the PresenZ player. The detection phase gathers information on what is visible from the ZOV, while the render phase applies lighting and shading to create colour data and other output variables. The PresenZ Merger/Prz\_maker transforms those "raw" images into a ".PRZ" file that can be viewed in the PresenZ player.


# Render blocks and occlusion factor

**The occlusion factor**

The concept of occlusion factor is directly linked to the concept of the Zone Of View.\
Simply put, it represents the quantity of objects that are hidden from the center of the Zone of View, but visible from another location in the Zone Of View.<br>

For instance, in a scene with rows of standing dominos, a lot of those might be invisible from the center of the ZOV because they are occluded by the first domino, but still, they can be seen from other parts of the ZOV.\
The same will happen in a forest with leaves and trunks.\
All those "occluded" objects must still be rendered for V-Nova PresenZ since they can be seen from the ZOV. In that case, we have a high occlusion factor.\
\
Now imagine the Zone Of View is in the middle of a big sphere made of concrete. In that extreme case, there is no object occluded; everything that is visible from the whole ZOV is also visible from the center of the ZOV, and the occlusion factor will be null.\
\
**The render blocks**\
\
The render phase breaks down the render into blocks.

* These blocks are all automatically rendered one after the other, but can be rendered separately if needed, in case a single block fails.
* All blocks are needed to cover the ZOV.&#x20;
* The number of blocks can vary from scene to scene, directly depending on the occlusion factor, how much the 3D scene covers the 360 view, or how close you get to some objects.
* Normally, there should only be 2 to 4 blocks.

{% hint style="info" %}
*Note:  There is a safety under "advanced rendering" that sets max blocks to 8 by default. Anything more, you would need to check whether there is a scale issue, a tagging issue, or a Zone of View issue causing the scene to need more blocks.*
{% endhint %}


# The Zone of View size

**Impact of changing the size of the Zone of View**

The size of the Zone of View (ZOV) plays a critical role in the V-Nova PresenZ experience. \
It represents the agency that the user will have in VR to\` move around.\
We strongly recommend staying with the default size ( 1m x 1m x 0.5m ). This covers all the natural movements in VR that the user will do while seated or standing in one spot.

A larger ZOV gives more agency to the viewer, but it usually quickly starts overwhelming the system during rendering and playback. Imagine a ZOV large enough to go into the next room; then all the information to cover the next room must also be computed, stored, and displayed.

There is only one case where the ZOV can be freely scaled without problem, which is when the Occlusion factor in the 3D scene is very low.  For example, a ZOV in the middle of the sky with mountains far away will have a very low occlusion factor, allowing for a very large ZOV of several meters in length without any issues.&#x20;

However, even in a scene with a high occlusion factor, it’s still possible to create a large space for the viewer to move around using multi-boxing. This technique involves creating [multiple ZOVs](https://support.presenzvr.com/docs/html/GeneralInformation/4_Multiboxing.html#multiboxing) located next to each other, providing a larger overall viewing area while maintaining a manageable occlusion factor.

In summary, it is usually best to keep the default size, and if necessary, use the multi-boxing approach if a larger overall viewing area is necessary.<br>

**The Zone of View size does not impact the scale of the Scene**

It’s important to note that changing the size of the Zone Of View will not change the perceived scale of the scene for the VR user; it will just change the "agency" the user has, i.e., how much the user moves in the scene. Notice in the animated gif below how the avatar in the UI, representing the size of the VR user, does not scale, while we change the ZOV size.<br>

<figure><img src="https://support.presenzvr.com/docs/html/_images/ZOVscalegif.gif" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
***Note:** By default, the Zone of View is 1m x 0.5m x 1m. If you want to have a standing experience with the virtual ground matching the real ground, it is recommended, if possible, to increase the ZOV height to at least 1 meter, to accommodate different people's heights.* <br>
{% endhint %}


# Render scale

**Size matters in VR**

When you’re in Virtual Reality, the size of objects around you can greatly affect your perception of the virtual world. Your eyes perceive depth and scale, so if the virtual world is too big, you will feel like a Lilliputian, and if it is too small, you will feel like a giant.&#x20;

**The Avatar in the UI**

To help you understand how big or small you will feel in the virtual world, we show you a representation of the VR user as an avatar in the interface. This will give you a clear idea of the size of objects around you.

**The render scale**

The render scale lets you scale the entire 3D scene during the PresenZ render. When you increase the render scale, it makes the virtual world appear larger in VR. For example, if you set the render scale to 2.0, the scene will appear twice as big in VR. \
But we are not really scaling or modifying the 3D scene in your software, because that will create a lot of problems. So to represent this change of scale, we modify both the visualization of the ZOV and the avatar in the UI.\
In the previous example, with a render scale of 2, this will shrink the viewer avatar by half in UI.\
And indeed, if you envision yourself wearing the VR headset and being the size of the UI avatar, the scene around you will be two times bigger.

Notice in the animated GIF below how both the avatar and the Zone Of View scale while changing the Render Scale.

<figure><img src="https://support.presenzvr.com/docs/html/_images/renderscalegif.gif" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*Note: It’s important to understand that the size of the zone of view is always in “real world” scale, so if it’s a 1 x 1 x 1 meter cube, the VR user will be able to move around freely in that 1 x 1 x 1 meter space. Changing the render scale won't modify that; it will just change the scale of the 3D scene rendered.*
{% endhint %}

**When to use the render scale**

The render scale is a powerful tool, especially when you’re working with virtual scenes that were not modeled using real-world units or need to be converted to meters for the PresenZ player.

Render scale can also be used as an artistic tool to make the VR user feel like a Lilliputian or a Giant

**Unit conversion:**

PresenZ will consider the scene as being in centimeters. So if render scale is set to 1.0, one unit in your authoring tool = one centimeter when seen in VR by the user.

&#x20;If you are using the **imperial units** system, you will have to adjust your render scale accordingly. The conversion would be 1 inch = 2.54 centimeters. So you should set the render scale to 2.54

\ <br>


# Deep reflections/stereo rendering

**Deep reflections concept**

V-Nova PresenZ includes a feature called “deep reflections,” which allows users to see different reflections for the left and right eye when wearing a VR headset. This feature is particularly useful for scenes that contain reflective surfaces.

By default, V-Nova PresenZ enables the deep reflection/ stereo parameter when rendering. This ensures that reflections appear to the VR user with "depth" as if they are reflecting the actual 3D scene.\
Without it, reflection will appear as "flat" on the surface of the reflective object.&#x20;

Therefore, unless you have a specific reason not to use it, it’s recommended that you always render your final images with the deep reflection parameter enabled.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FY6wxYnizufMphnTgoY5D%2F5.jpeg?alt=media)

In the image above, which was rendered with deep reflections, you can see how the left and right eye reflections are slightly different. This helps to create a proper sense of depth in the reflections, contributing to a more immersive VR experience.


# Transparencies

**Transparencies**

**Standard transparency shading is "baked" on the surface**

By default, V-Nova PresenZ will not go through a transparent object in the detection phase.\
This will end up baking what is seen through the transparent object onto its surface.\
For instance, a window surface will feel like a "poster" of the background.

By default, the baked on the surface combined with Deep reflection/ stereo works quite well for complex refractive objects like the water drop, a frosted window, a vase, etc., because the objects’ curvature distorts most of the refractions. It also works well for faraway windows.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FqRIIc1DatBNTxiCujNFS%2F6.png?alt=media) ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F4SUOYu6kfumNQc72K4TC%2F7.jpeg?alt=media)

**When to use Glass tagging**

But for close and flat transparent surfaces like a nearby windows, this poster effect will be weird.\
We need to have parallax on what is seen behind the window.\
\
Tagging the transparent surface as "Glass" will solve the issue.\
With this tag, V-Nova PresenZ will keep the reflection component of the tagged surface and start to scan and capture what is behind it. This cancels the poster effect because PresenZ then captures in 3D what is behind the glass-tagged surface in the 3D scene.

Some examples below where glass tagging the nearby transparent objects is mandatory

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FEIk33okG0A2SHSfe4lNq%2F8.png?alt=media) ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F3bBkko9LFUWvK6kO0SzP%2F9.png?alt=media)


# The specular point

**The specular point**

To create a volumetric image in V-Nova PresenZ, it’s necessary to render from different locations within the Zone of View (ZOV), unlike a regular perspective render that’s taken from a single point. When rendering reflective surfaces such as mirrors, V-Nova PresenZ might elect to render parts of the same mirror from different viewing points. If unmodified, this would lead to a big difference in appearance for those different parts because they will each show a reflection seen from a different viewpoint.

To ensure that the reflections appear consistent across all parts of the mirror, V-Nova PresenZ “bends” the reflection rays as if they were all computed from a single point. This point is called the specular point and is represented by the point in between the viewer avatar’s eyes. It can be moved within the PresenZ user interface to adjust the position of the point from which the reflections are computed. It is good practice to usually let the specular point be at the center of the zone of view.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FXsf4srwJB7AkFAC7aIWA%2F13.gif?alt=media)

{% hint style="warning" %}
***Warning:** It’s important to note that if the user moves too far away from the specular point while viewing the rendered image in VR, reflections may start to appear distorted or strange. For multiboxing, it’s generally best to locate the specular point in the central ZOV to ensure that reflections remain consistent across all ZOVs.*
{% endhint %}


# Rendering resolution

&#x20;**Rendering resolution**

Currently in our Authoring released Plugins, V-Nova PresenZ imposes some limits in the choice of resolutions and aspect ratios we can use to render.

* Our volumetric image format (.PRZ) is able to handle image resolution going from (6048,4032) to (288, 192). Usually we render at (3024,2016), which corresponds to around 4K to 6K, equivalent to a  360 image
* The PresenZ camera uses a cubeMap3x2 projection, which brings us to lock the aspect ratio to 1.5
* For technical reasons, mostly linked to the underscanning system, the resolution must be a multiple of 24.

To summarize, the rendering resolution must be a multiple of 3x24=72 in width and 2x24=48 in height, with a width-to-height ratio of 1.5, and must be within the range of (6048, 4032) to (288, 192). Here are some examples of supported resolutions:

| (288 ,192 ) Minimum       | 4 x(72) | 4 x(72x3/2) |
| ------------------------- | ------- | ----------- |
| (720 ,480 )               | 10x(72) | 10x(72x3/2) |
| (1512,1008) Good for Test | 21x(72) | 21x48       |
| (2160,1440)               | 30x(72) | 30x(72x3/2) |
| (3024,2016) Default       | 42x(72) | 42x(72x3/2) |
| (6048,4032)               | 84x(72) | 84x(72x3/2) |

{% hint style="info" %}
*Note: Through their graphical interface, our authoring plugins will help you to set a compatible resolution.*
{% endhint %}

<img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FftxiMlyykeYrtwe150Hn%2FScreenshot%202025-08-26%20140748.png?alt=media&amp;token=c5118ebd-27ce-4f0d-88a7-f53f58150a5a" alt="" data-size="original">\
\
***Warning:** Unless you really know what you are doing, we do not recommend going higher than the default: 3024 \* 2016*


# Chaotic tag: rendering grass, hairs, fur, trees...

[<br>](https://support.presenzvr.com/docs/html/_images/InsideZOV.png)**When to use the chaotic tag:**

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FjCXHImvJ2muactMVb63S%2F15.png?alt=media)

If you’re working on a scene with complex geometry, such as vegetation or hair, you can speed up your rendering process hugely by using the PresenZ Chaotic tag. This tag reduces the number of unnecessary points rendered for complex objects, decreasing the overall render time and the size of the final .prz file&#x20;

**How does it work?**

To achieve this, V-Nova PresenZ divides the space around the Zone of View into smaller volumes corresponding to 8x8 pixel patches, with a variable depth. \
As the distance from the center increases, the slices become deeper, as we don’t require as much detail further away from the ZOV.\
\
This is a top view representing the space partition:

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FGkoufqgyvLeCN4zPeb5d%2F16.jpeg?alt=media)

For each patch, V-Nova PresenZ computes a maximum of 8x8 pixels (64 pixels) instead of the original complexity of your objects. Without this process, leaves or grass could generate hundreds of points for the same patch volume. This approach can significantly reduce the complexity of your scene and speed up rendering.&#x20;

We provide a UI to tag as "chaotic" selected objects in your 3D scene.


# Draft mode

**Draft mode**

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fm6vrf6VhU8KSHd0W3ZGF%2Fimage.png?alt=media&amp;token=1462df13-791a-48cc-9f4f-ef9368b949b3" alt=""><figcaption></figcaption></figure>

Draft mode is a special debug/test mode of PresenZ that can be super useful for quick tests:

* It does a volumetric rendering, so depth information is stored for every point
* But only from the center of the Zone Of View

So, of course, any occluded object not visible from the center of the ZOV won't be captured but:

* It renders very quickly, normally almost like a 2d image,
* The result still allows to test in VR the scene composition, scale, animation, and placement of the Zone of View.

{% hint style="info" %}
[*Note: We recommend when working on a new scene to start in draft mode low resolution to iterate quickly.*](#user-content-fn-1)[^1]
{% endhint %}

[^1]:


# Underscan

**Closer = more defined = more points**

In V-Nova PresenZ, the amount of data collected for a surface depends on the surface’s distance from the ZOV. The closer a surface is to the ZOV, the more points are needed to represent it accurately in the rendered image. Therefore, the point density will be higher for closer surfaces compared to surfaces further away. Which is generally desirable.<br>

**When Underscanning is needed**

But when getting too close to the ZOV, this process might start to generate too many details.

The underscanning system is designed to reduce the amount of data collected for surfaces very close to the ZOV, while still maintaining an acceptable level of accuracy. It aims to keep the collected data above a minimum point size for optimal display in a VR headset.

The default setting for the minimum point size is 1mm, <br>

{% hint style="info" %}
**Note:** It should be noted that if a scene has objects very close to the ZOV and the rendering times and file sizes are too high, increasing the minimum point size might be beneficial. The larger the point size, the more underscanning will be triggered.
{% endhint %}


# Rendering inside the ZOV

**Rendering inside the ZOV**

The PresenZ rendering technique can also be used to render objects inside the Zone of View (ZOV) instead of outside it. This essentially inverts the ZOV, allowing you to present objects in 3D space and providing the viewer with the freedom to walk around it and view the object from every possible angle. This feature is particularly useful for presenting products or conceptual designs and can create a more captivating experience by allowing objects to traverse the ZOV seamlessly.

Rendering inside the ZOV also enables the implementation of Augmented Reality (AR) applications. However, it’s important to note that rendering inside the ZOV increases the computational complexity of the scene, which can result in longer rendering times. Therefore, we recommend enabling this feature only when there is scene geometry located inside the ZOV.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fx39qtCE80kgriKNFa7FG%2Fimage.png?alt=media&amp;token=df47dcd0-876f-4d13-8940-b22b02b3b133" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
**Warning:** For a standard, "inside out" PresenZ rendering, when the viewer is supposed to be inside the zone of view and the interesting objects are outside. We recommend to avoid any object to be inside the zone of view. The viewer might end up inside an object, and overall the load of scene might be heavily impacted.
{% endhint %}


# The clipping sphere

&#x20;**The clipping sphere**

The clipping sphere feature in V-Nova PresenZ allows you to render only a specific part of a 3D scene. This can be useful if you’ve made changes to a particular object and want to see how those changes have affected it without having to render the entire scene. Instead, you can place a clipping sphere around the object and render only that part. This is similar to doing a cropped render in 2D.

In the rendered image, everything outside of the clipping sphere is not visible, giving the impression that the object is isolated in space. In some cases, you might want to do the opposite and render everything outside of the sphere, creating a hole in the final render.

This technique can also be used for animations where a large part of the scene is static and unaffected by the animation (such as shadows and reflections). By rendering this static part of the scene once using the clipping sphere feature, usually outside the sphere, you can save time and resources during the animation rendering process, by just rendering inside the sphere for the animation.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F3OOAWordhgNfQaUzas13%2F14.jpeg?alt=media)


# Smoke, fire and atmospheric rendering

How to render Atmospheric volumes

#### Smoke, fire and atmospheric rendering&#x20;

V-Nova PresenZ does support Smoke, Fire, atmospheric and more broadly "volumetric" rendering from Maya Arnold. But it is important to remember that a volume that takes up a large space will greatly increase the number of points and the amount of data in the scene. So this will take up more blocks in the render. As such, you should limit the amount of space used for atmospheric volumes by using the aiAtmosphereVolume shader on a geometry. As seen below

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FDZ7BQOym5RnDJRugaoDP%2Fimage.png?alt=media&amp;token=76a41979-f8d2-415a-ad5d-3316df538866" alt=""><figcaption></figcaption></figure>

You should avoid the atmosphere attribute inside Arnold, as this will create points throughout the scene and will affect your renders, as mentioned. It will also create issues where you will need to render inside the Zone Of View.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F4uTygkQaT5ZVvobEv6wr%2Fimage.png?alt=media&amp;token=9be63ae9-4799-4d68-9694-3724797558c1" alt=""><figcaption></figcaption></figure>

When rendering any volume, it is important to limit the number of effective Lights and improve the samples of the render inside of Arnold

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FYtaOLhZJLmHKbhufL0SK%2Fimage.png?alt=media&amp;token=1e47ed0f-d469-4cee-a9dc-f35f3c1e3e24" alt=""><figcaption></figcaption></figure>

* Here is a simple look at render settings that will work for testing

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FEcVC30oDHIniNR2n5qIz%2Fimage.png?alt=media&amp;token=b60c541a-d667-40cc-95a0-d1d81ef5e2e8" alt=""><figcaption></figcaption></figure>


# File formats and structures in V-Nova PresenZ

Understanding the different files utilized by V-Nova PresenZ

#### **File formats in V-Nova PresenZ**&#x20;

**.przDetect**&#x20;

Detect files are generated in the first section of the two-phase render process in V-Nova PresenZ. They contain information on what is visible from inside the Zone Of View. This is an essential part of the render and limits the information that is rendered in each block. It contains info on geometry but no color data or any aov’s. It can still be inspected inside Houdini with the V-Nova PresenZ nodes. And it is used for the second phase: the render phase.

* 2d equivalent: none, this file is a technical intermediary file

**.przRender**

This is an uncompressed file format used for the compositing phase of V-Nova PresenZ. It contains all the aov’s and colour values useful in the compositing process. This is a large file that is not suitable for viewing in playback.&#x20;

* 2d equivalent:  “.exr” or ".raw"&#x20;

**.prz** =  these files are usually only compressed in lossless. They are PresenZ Image generated by the Merge/Prz\_Maker from the .przRender file.  It’s not a massive compression on the file, but it can be used in the viewer or player to have effective playback.&#x20;

* 2d equivalent: “.png”

**.przShot**

A lossy but qualitative compression of a sequence of .prz files. This is much more compact and convenient to carry around. It can be directly played back by the player.&#x20;

* 2d equivalent:  a ".mov" corresponding to one shot of the movi&#x65;**.**

\
**.przToc**

A file that defines the PresenZ movie, pointing to the different shots, audio, background, and other attributes specific to PresenZ. The TOC stands for Table Of Content.\
It is usually generated by the Sequencer, the equivalent of a Video Editing software.

* 2d equivalent:  ".edl" <br>


# Motion vectors and animation

**Motion vectors and animation**

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FHafmiMEDLCwfXSMStrCQ%2F17.png?alt=media)

**Motion vectors** are used by V-Nova PresenZ to smooth the motion of animated objects in the volumetric image at display time. They hold local information about the motion of each point for each frame. The motion vectors can be computed by most rendering engines and can be incorporated into a compositing software.&#x20;

V-Nova PresenZ requires a specific type of motion vector that describes the trajectory of objects from the current frame to the next frame. Basically for each point: motion vector(frame) = position  (frame+1) - position (frame)

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FJZYYe3n4DJ4OxiN5DSr8%2F18.png?alt=media)

{% hint style="info" %}
***Note:** Our authoring plugins can detect when the rendering is in an animation context and will prompt the user to apply the motion vector settings automatically.*
{% endhint %}

{% hint style="warning" %}
**Warning:** Attention, don't confuse the motion vector concept with the "velocity" in simulation dynamics.\
The motion vector we are looking for is the vector pointing to the position at the next frame\
, while the velocity is the current speed of the object at the beginning of the frame simulated before any external force is applied (gravity, wind, etc).&#x20;
{% endhint %}


# Normals inside V-Nova PresenZ

&#x20;**Normals inside V-Nova PresenZ**

In the 3D industry, there are many different workflows and programs, and when you compile all your geometry together, it’s possible for normals to end up reverted. While this can cause limited issues in a standard perspective render, the implications can be more severe when using V-Nova PresenZ.

During the detection phase, V-Nova PresenZ uses intensively surface normals. If the normals are incorrect, it can lead to under- or over-defined surfaces, which can cause major Moiré effects.

To address this issue, it’s recommended that you check your normals using a quick normal shader render if you encounter problems such as major Moire effects. This can help identify any incorrect normals that may be causing issues with your PresenZ renders.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F1q86VsWcIs7FGmiLf3ya%2F12.jpeg?alt=media)


# Quick tips

**Quick tips**

Some miscellaneous helpful tips for V-Nova PresenZ

* **Camera:** Make sure your camera is visible to V-Nova PresenZ; don’t put it on a hidden layer. Also, be sure the camera is not locked and has its parameters editable.
* **Quick tests:** Remember that you can change the renderer resolution of V-Nova PresenZ to perform quick tests. Don’t be afraid to use 1200x800 or even 720x480 to make quick test renders. It can be combined with Draft Mode.
* **Command line merger/Prz\_Maker:** When using the merger.exe to merge your rendered files ( .przRenders ) into a .PRZ file for the Viewer. Make sure that the directory of the merger is your current directory, so it can find the necessary files.


# V-Nova PresenZ GUI for Maya and Arnold

#### PresenZ Plug-In Maya and Arnold

When you have successfully installed V-Nova PresenZ and launch Maya, a new shelf should be added named “PresenZArnold”. If you open the shelf, you will see a toolbar appear with 7 buttons.

<figure><img src="https://support.presenzvr.com/docs/html/_images/FloatingToolbar.png" alt=""><figcaption></figcaption></figure>

From left to right:

* [The render dialog](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#renderdialogmayaarnold): an easy way to launch a render.
* [Create Zone Of View](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#createzovmayaarnold): create a Zone Of View at the center of the scene.
* [The Tagging Manager](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#taggingmanagermayaarnold): to tag a surface with the PresenZ glass attribute settings, or to tag an object as a prz\_chaotic
* [The Window Render Shape Creation dialog](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#arnoldmaya-windowmeshgui): Open a Dialog to create the PresenZ Window Rendering Shape from a polygon mesh.
* PresenZ maker UI: This allows you to manually create .prz files by inputting file paths and directories, and V-Nova PresenZ runs the command line Merge tool inside of Maya.
* Help: Will direct you to the V-Nova PresenZ support page.
* Updater: Updates V-Nova PresenZ to the latest version.

Note

If, for some reason, you do not see this toolbar in Maya, see the [troubleshooting section](https://support.presenzvr.com/docs/html/ArnoldMaya/Troobleshooting.html#mayaarnoldtroobleshootingtb)

### The render dialog

<figure><img src="https://support.presenzvr.com/docs/html/_images/RenderDialogPlug.png" alt=""><figcaption></figcaption></figure>

List all the available Zone Of View in your scene and allow the [rendering](https://support.presenzvr.com/docs/html/ArnoldMaya/Rendering_With_PresenZ_Arnold.html#renderingwithpresenzmayaarnold) of each individually.

In case you did not set certain settings, such as setting the render engine to Arnold, enabling bucket rendering, or creating a ZOV, The Dialog will prompt you to automatically change these settings.

<figure><img src="https://support.presenzvr.com/docs/html/_images/PresenzApplyChanges.PNG" alt=""><figcaption></figcaption></figure>

### Creating a Zone Of View

Use the Icon to create a Zone Of View in your scene.

![../\_images/presenzcamera.PNG](https://support.presenzvr.com/docs/html/_images/presenzcamera.PNG)

Once a camera is created in your viewport, it should appear like this:

<figure><img src="https://support.presenzvr.com/docs/html/_images/BigUIShot.PNG" alt=""><figcaption></figcaption></figure>

As you can see, the camera is represented by a box ( the Zone Of View ), with a viewer avatar inside it. The viewer avatar represents the VR user; its orientation corresponds to the default orientation of the VR user. This UI allows you to visualize the effects of changing parameters, like the size of the zone of view, the render scale, or the viewer's IPD.

### Zone Of View parameters

The PresenZ Camera Parameter section is divided into 6 sections.

<figure><img src="https://support.presenzvr.com/docs/html/_images/parameter.png" alt=""><figcaption></figcaption></figure>

* **PresenZ enabled**: On by default. Uncheck this box to disable the PresenZ render.
* **Render scale**: Default is 1.0. This will change the scale of the scene in VR without having to rescale all objects in the Maya scene.
* **Render isolated**: Only render objects that have been tagged as Isolated [with the Tagging Manager.](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#taggingmanagermayaarnold)

#### Zone of View dimensions

<figure><img src="https://support.presenzvr.com/docs/html/_images/ZOVDimensions.PNG" alt=""><figcaption></figcaption></figure>

* **Zone of View dimensions**: By default, the Zone of View is 1.0m, 1.0m, 0.5m, and the distance to the ground is 1.6m. These can be changed to increase the size of the Zone of View. [For more information on the Zone of View, please refer to this section](https://support.presenzvr.com/docs/html/GeneralInformation/1_The_zone_of_view.html#zov).

#### Ground Settings

<figure><img src="https://support.presenzvr.com/docs/html/_images/GroundSettings.PNG" alt=""><figcaption></figcaption></figure>

* **Display ground**: On by default. Allows to visualize how far the center of the ZOV will be from the ground in real life for the VR user. Ideally, you want it to match the ground in the 3D scene, so that the VR user will feel like they are standing on the ground and not floating.

#### Rendering Options

<figure><img src="https://support.presenzvr.com/docs/html/_images/RenderingOptions.PNG" alt=""><figcaption></figcaption></figure>

#### **Draft Rendering**

&#x20;Off by default. Enables PresenZ draft rendering mode. This feature allows you to make a quick VR test render. This is helpful to have a sense of your lighting setup, position of the [Zone Of View](https://support.presenzvr.com/docs/html/GeneralInformation/1_The_zone_of_view.html#zov), and the general scale of your scene. Without having to do a full render. But this will not cover all the possible viewing positions inside the ZOV; you will see “undefined” or missing zones with the Draft Rendering enabled, so it is not suitable for production.

* **Render inside ZOV**: Off by default. This enables the rendering of objects inside the ZOV. Unless checked, objects within the ZOV won’t be rendered.
* **Render isolated**: Only render objects that have been tagged as Isolated [with the Tagging Manager.](https://support.presenzvr.com/docs/html/ArnoldMaya/ArnoldPluginMaya.html#taggingmanagermayaarnold)

By default, V-Nova PresenZ will render a window as a “stereoscopic poster” displaying what can be seen behind the window. This works great for far-away windows or transparent objects, but for objects located close by to the ZOV, it is recommended to tag them as “glass” in order for V-Nova PresenZ to render them correctly. More information on transparency and glass objects can be found [here](https://support.presenzvr.com/docs/html/GeneralInformation/7_Transparency.html#transparency)

## Rendering

### Output Render Path

Before rendering, you should specify the exact path to where you want to save the .PRZ image. In the Maya Render Settings menu, go to the Common tab and enter the full file path in the File Name Prefix and select “prz” as the Image Format. An example of a full file path would be: “C:\path\to\renders\yourFileName”.

V-Nova PresenZ will also create a przDetect and przRender folder at that location. These folders are used to store intermediate files generated by the PresenZ [two-phase system](https://support.presenzvr.com/docs/html/GeneralInformation/2_The_Two_Phases_System.html#twophasessystem).

<figure><img src="https://support.presenzvr.com/docs/html/_images/FullFilePath2.png" alt=""><figcaption></figcaption></figure>

### Detecting, rendering and merging

To render your PRZ image, select the PresenZ render button.

<figure><img src="https://support.presenzvr.com/docs/html/_images/RenderDialogRender1.png" alt=""><figcaption></figcaption></figure>

Be sure to have properly set your output location, as explained in the [previous section](https://support.presenzvr.com/docs/html/VrayMaya/Rendering_With_PresenZ.html#outputrenderpath-maya).

* **Zone of View**: allows you to select which Zone of View you are going to render (Since you can have several in your 3D Scene).
* **Merge files after rendering**: On by default. This will automatically start the merger after completing the different phases of the rendering process. Alternatively, you can manually merge your output files through the command line. [This is explained in the PresenZ Utility Merger section](https://support.presenzvr.com/docs/html/Utilities/PresenZMergerUtility.html#presenzmergerutility).
* **Render button**: This button will automatically launch the different [PresenZ render phases](https://support.presenzvr.com/docs/html/GeneralInformation/2_The_Two_Phases_System.html#twophasessystem), one after the other, according to your [zov settings](https://support.presenzvr.com/docs/html/VrayMaya/VrayPluginMaya.html#advancedphasesmayavray). Its name will also change according to your settings. “Detect render merger”, for instance.
* **Merge button**: This button allows you to manually re-merge your .przRender files after they have been rendered.
* **Progress bar**: Depending on the options, the whole PresenZ render will take several phases and possibly passes (for instance, opaque and glass passes). This will show you the progress in those phases/passes.
* **Detection safeguard**: Located in the Advanced section. This is on by default and set to 8 blocks. 8 blocks should be plenty to handle very complex scenes properly set. If the detection phase issues more than 98 blocks, usually it means your scene has complex geometry not tagged as Chaotic. [Visit this section to reduce the number of blocks](https://support.presenzvr.com/docs/html/VrayMaya/VrayPluginMaya.html#taggingmanagermayavray).

{% hint style="info" %}
Note: The merge button is useful to apply your [V-Ray Frame Buffer Color Correction Settings](https://support.presenzvr.com/docs/html/VrayMaya/Vray_Specifics.html#vfb-colorcorrectionsmaya) without having to re-render.
{% endhint %}

{% hint style="info" %}
Note: Near the progress bar a number will indicate how many passes will be executed for your render; this number can change. For a normal render, this number will be 3: a detection pass, a render pass, and the merge. As another example, if you have decided to render in “opaque and glass” mode, the whole PresenZ render will take more passes than a standard render. The total number of passes will be 5: one detection pass and render pass for the opaque objects, one detection and render pass for the transparent objects, and a final merge.
{% endhint %}


# Getting Started with V-Nova PresenZ Maya and Arnold

A brief and basic starting tutorial with some instruction on how to get started with V-Nova PresentZ in Maya and Arnold

{% embed url="<https://vimeo.com/1152191707?fe=ci&fl=sv&share=copy>" %}

**V-Nova PresenZ Maya and Arnold**

This section is broken up into key categories that may require a bit more explanation. along with a hands-on tutorial to show how we deal with this in our software. Please read through the [PresenZ Plug-In Maya and Arnold](/technologies/v-nova.presenz/getting-started/v-nova-presenz-gui-for-maya-and-arnold) section to go through the addon and shelf tools before proceeding.

* Getting Started with V-Nova PresenZ Maya and Arnold

  * [Render Scale](#render-scale)
  * [Resolution and Formats](#resolution-and-format)
  * [Motion Vectors](#motion-vectors)
  * [Tagging - Transparency](#tagging-transparency)
  * [Tagging - Chaotic](#tagging-chaotic-objects)
  * [Atmospheric Rendering](#atmospheric-rendering)
  * [Clipping Sphere ](#clipping-sphere)

## Render Scale

{% embed url="<https://vimeo.com/1152244634?fe=ci&fl=sv&share=copy>" %}

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FmXKBQ5kjooDAKCpY3vyE%2Fimage.png?alt=media&amp;token=f838dcae-91da-4c38-b9d0-4b8d0953e33b" alt=""><figcaption></figcaption></figure>

Render scale is the most basic requirement for V-Nova PresenZ; it controls the feel of the whole scene and will allow us to feel small or large in that environment. Setting this scale to the correct size to match the human in VR will mean you can interact and move in the environment accurately.&#x20;

You should use the Render Scale attribute to match the scene scale. This can be done by typing in a value or using the "Ctrl + left mouse " to drag the value

* Using the incorrect scale can negatively effect you render and increase the number of blocs dramatically increasing render time.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FCXYXQyiPK178fOZ8VYfy%2Fimage.png?alt=media&amp;token=3d80f958-010a-4885-8862-c6a5e554c846" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fgv0INMAhdClgLXj3K1rU%2FScreenshot%202025-09-09%20094438.png?alt=media&amp;token=7bb5382c-21a5-4038-bab9-36ead614ff9d" alt="" width="375"><figcaption></figcaption></figure>

Using the Zone of view dimensions is not advised, but these can be used to enlarge or minimise the available head movement inside VR. This attribute greatly increases render time and the amount of data created. So any editing should be limited.

You should always try to keep the avatar at the center of the Zone of View to the correct human scale in your scene, as shown below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FybVV1FF8YqyeBVFq0Tfp%2Fimage.png?alt=media&amp;token=630141fd-13be-44e1-b3b3-30d32e7f11ec" alt=""><figcaption></figcaption></figure>

**Size matters in VR**

When you’re in Virtual Reality, the size of objects around you can greatly affect your perception of the virtual world. Your eyes perceive depth and scale, so if the virtual world is too big, you will feel like a Lilliputian, and if it is too small, you will feel like a giant.&#x20;

**The Avatar in the UI**

To help you understand how big or small you will feel in the virtual world, we show you a representation of the VR user as an avatar in the interface. This will give you a clear idea of the size of objects around you.

**The render scale**

The render scale lets you scale the entire 3D scene during the PresenZ render. When you increase the render scale, it makes the virtual world appear larger in VR. For example, if you set the render scale to 2.0, the scene will appear twice as big in VR. \
But we are not really scaling or modifying the 3D scene in your software, because that will create a lot of problems. So to represent this change of scale, we modify both the visualization of the ZOV and the avatar in the UI.\
In the previous example, with a render scale of 2, this will shrink the viewer avatar by half in UI.\
And indeed, if you envision yourself wearing the VR headset and being the size of the UI avatar, the scene around you will be two times bigger.

Notice in the animated GIF below how both the avatar and the Zone Of View scale while changing the Render Scale.

<figure><img src="https://support.presenzvr.com/docs/html/_images/renderscalegif.gif" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*Note: It’s important to understand that the size of the zone of view is always in “real world” scale, so if it’s a 1 x 1 x 1 meter cube, the VR user will be able to move around freely in that 1 x 1 x 1 meter space. Changing the render scale won't modify that; it will just change the scale of the 3D scene rendered.*
{% endhint %}

**When to use the render scale**

The render scale is a powerful tool, especially when you’re working with virtual scenes that were not modeled using real-world units or need to be converted to meters for the PresenZ player.

Render scale can also be used as an artistic tool to make the VR user feel like a Lilliputian or a Giant

**Unit conversion:**

PresenZ will consider the scene as being in centimeters. So if render scale is set to 1.0, one unit in your authoring tool = one centimeter when seen in VR by the user.

&#x20;If you are using the **imperial units** system, you will have to adjust your render scale accordingly. The conversion would be 1 inch = 2.54 centimeters. So you should set the render scale to 2.54

## Resolution and Format

&#x20;**Rendering resolution**

Currently in our Authoring released Plugins, V-Nova PresenZ imposes some limits in the choice of resolutions and aspect ratios we can use to render.

* Our volumetric image format (.PRZ) is able to handle image resolution going from (6048,4032) to (288, 192). Usually we render at (3024,2016), which corresponds to around 4K to 6K, equivalent to a  360 image
* The PresenZ camera uses a cubeMap3x2 projection, which brings us to lock the aspect ratio to 1.5
* For technical reasons, mostly linked to the underscanning system, the resolution must be a multiple of 24.

To summarize, the rendering resolution must be a multiple of 3x24=72 in width and 2x24=48 in height, with a width-to-height ratio of 1.5, and must be within the range of (6048, 4032) to (288, 192). Here are some examples of supported resolutions:

| (288 ,192 ) Minimum       | 4 x(72) | 4 x(72x3/2) |
| ------------------------- | ------- | ----------- |
| (720 ,480 )               | 10x(72) | 10x(72x3/2) |
| (1512,1008) Good for Test | 21x(72) | 21x48       |
| (2160,1440)               | 30x(72) | 30x(72x3/2) |
| (3024,2016) Default       | 42x(72) | 42x(72x3/2) |
| (6048,4032)               | 84x(72) | 84x(72x3/2) |

{% hint style="info" %}
*Note: Through their graphical interface, our authoring plugins will help you to set a compatible resolution.*
{% endhint %}

<img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FftxiMlyykeYrtwe150Hn%2FScreenshot%202025-08-26%20140748.png?alt=media&amp;token=c5118ebd-27ce-4f0d-88a7-f53f58150a5a" alt="" data-size="original">\
\
***Warning:** Unless you really know what you are doing, we do not recommend going higher than the default: 3024 \* 2016*

### Formats and file types

V-Nova PresenZ utilises a variety of file formats to both create files in uncompressed format for use in comp and in compressed format for viewing in a player. This is a quick run-through of the common formats that you will see, along with an image to show the standard file structure created when you render from Maya.

&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F1gM7XxMiDNczyJfoCTVe%2Fimage.png?alt=media&amp;token=6afc2d49-95c6-49df-ac7b-511649bdf84e" alt=""><figcaption></figcaption></figure>

#### **File formats in V-Nova PresenZ**&#x20;

**.przDetect**&#x20;

Detect files are generated in the first section of the two-phase render process in V-Nova PresenZ. They contain information on what is visible from inside the Zone Of View. This is an essential part of the render and limits the information that is rendered in each block. It contains info on geometry but no color data or any aov’s. It can still be inspected inside Houdini with the V-Nova PresenZ nodes. And it is used for the second phase: the render phase.

* 2d equivalent: none, this file is a technical intermediary file

**.przRender**

This is an uncompressed file format used for the compositing phase of V-Nova PresenZ. It contains all the aov’s and colour values useful in the compositing process. This is a large file that is not suitable for viewing in playback.&#x20;

* 2d equivalent:  “.exr” or ".raw"&#x20;

**.prz** =  these files are usually only compressed in lossless. They are PresenZ Image generated by the Merge/Prz\_Maker from the .przRender file.  It’s not a massive compression on the file, but it can be used in the viewer or player to have effective playback.&#x20;

* 2d equivalent: “.png”

**.przShot**

A lossy but qualitative compression of a sequence of .prz files. This is much more compact and convenient to carry around. It can be directly played back by the player.&#x20;

* 2d equivalent:  a ".mov" corresponding to one shot of the movi&#x65;**.**

\
**.przToc**

A file that defines the PresenZ movie, pointing to the different shots, audio, background, and other attributes specific to PresenZ. The TOC stands for Table Of Content.\
It is usually generated by the Sequencer, the equivalent of a Video Editing software.

* 2d equivalent:  ".edl" <br>

## Motion Vectors

{% embed url="<https://vimeo.com/reviews/aae4bc90-d5e1-487e-994c-41038dec7b0e/videos/1152244421>" %}

{% embed url="<https://vimeo.com/1152244421?fe=ci&fl=sv&share=copy>" %}

Motion Vectors are used by V-Nova PresenZ to interpolate / smooth transitions between frames in VR. Setting the values correctly to achieve this is vital. To get Motion Vectors, we need to set the scene to create motion blur, this give are points there correct data so that V-Nova PresenZ can interpolate.&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Frnecq5wBrSXkeFyfkBSB%2Fimage.png?alt=media&amp;token=696c997c-4a2f-4d0e-b34a-dab6526b2b1c" alt=""><figcaption></figcaption></figure>

We can then use the player to check these motion Vectors by using the speed function inside the playback terminal.&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FHYs6aTEXGhC7AVhBKAtR%2Fimage.png?alt=media&amp;token=8b0517f3-1c59-4226-9152-88fad4fcad14" alt=""><figcaption></figcaption></figure>

**Motion vectors and animation**

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FHafmiMEDLCwfXSMStrCQ%2F17.png?alt=media)

**Motion vectors** are used by V-Nova PresenZ to smooth the motion of animated objects in the volumetric image at display time. They hold local information about the motion of each point for each frame. The motion vectors can be computed by most rendering engines and can be incorporated into a compositing software.&#x20;

V-Nova PresenZ requires a specific type of motion vector that describes the trajectory of objects from the current frame to the next frame. Basically for each point: motion vector(frame) = position  (frame+1) - position (frame)

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FJZYYe3n4DJ4OxiN5DSr8%2F18.png?alt=media)

{% hint style="info" %}
***Note:** Our authoring plugins can detect when the rendering is in an animation context and will prompt the user to apply the motion vector settings automatically.*
{% endhint %}

{% hint style="warning" %}
**Warning:** Attention, don't confuse the motion vector concept with the "velocity" in simulation dynamics.\
The motion vector we are looking for is the vector pointing to the position at the next frame\
, while the velocity is the current speed of the object at the beginning of the frame simulated before any external force is applied (gravity, wind, etc).&#x20;
{% endhint %}

## Tagging Transparency

{% embed url="<https://vimeo.com/1152244873?fe=ci&fl=sv&share=copy>" %}

Rendering Transparent objects in V-Nova PresenZ by default will bake the transparent image from behind the object onto the surface. This reduces render time and, in most cases, will be unnoticeable. But for objects or surfaces that don't have much distortion, this can break emersion and cause a poster-type effect.&#x20;

To deal with this, you can use the Glass tag under the Tagging Manager ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FpConqje37mNQ5tqLBU1i%2Fimage.png?alt=media\&token=b0cbb4da-ff9e-4dd6-b21c-4a69ddda0e50)

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FexkS9D2tztPtkkdpiJnV%2FScreenshot%202025-08-05%20170923.png?alt=media&amp;token=657c77a7-b8b8-4041-b49d-abba5f0163e5" alt=""><figcaption></figcaption></figure>

* You can apply tags by simply selecting the geometry and hitting "Refresh" in the Tagging Manager
* Now simply left-click in the Glass box and hit "Apply changes."&#x20;

Now, during the detection phase of the render, V-Nova PresenZ will scan what is behind the object and give accurate details for anything visible.

**Transparencies**

**Standard transparency shading is "baked" on the surface**

By default, V-Nova PresenZ will not go through a transparent object in the detection phase.\
This will end up baking what is seen through the transparent object onto its surface.\
For instance, a window surface will feel like a "poster" of the background.

By default, the baked on the surface combined with Deep reflection/ stereo works quite well for complex refractive objects like the water drop, a frosted window, a vase, etc., because the objects’ curvature distorts most of the refractions. It also works well for faraway windows.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FqRIIc1DatBNTxiCujNFS%2F6.png?alt=media) ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F4SUOYu6kfumNQc72K4TC%2F7.jpeg?alt=media)

**When to use Glass tagging**

But for close and flat transparent surfaces like a nearby windows, this poster effect will be weird.\
We need to have parallax on what is seen behind the window.\
\
Tagging the transparent surface as "Glass" will solve the issue.\
With this tag, V-Nova PresenZ will keep the reflection component of the tagged surface and start to scan and capture what is behind it. This cancels the poster effect because PresenZ then captures in 3D what is behind the glass-tagged surface in the 3D scene.

Some examples below where glass tagging the nearby transparent objects is mandatory

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FEIk33okG0A2SHSfe4lNq%2F8.png?alt=media) ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F3bBkko9LFUWvK6kO0SzP%2F9.png?alt=media)

## Tagging Chaotic Objects

{% embed url="<https://vimeo.com/1152245800?share=>" %}

Chaotic objects are any objects that have a complex geometric shape that may need multiple scans (Detection phase) to correctly gather all the information for the Zone of View. In other words, if we move inside the ZOV, do we see more or less geo, is some of it occluded or made visible, and therefore must be scanned to correctly render.&#x20;

This process can be affected by many of the parameters inside the render scale and the ZOV scale, but is most predominant with Vegetation and Groom. we there for have the Chaotic tag inside the Tagging manager so that we can reduce the amount of information gathered without affecting the render quality.

The way to create a chaotic tag is by using Tagging Manager ![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FpConqje37mNQ5tqLBU1i%2Fimage.png?alt=media\&token=b0cbb4da-ff9e-4dd6-b21c-4a69ddda0e50)

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FD7BSVk52DDLgvBo5azkZ%2Fimage.png?alt=media&amp;token=4a137857-c773-45c1-9939-10b3bf243ecf" alt=""><figcaption></figcaption></figure>

This will compress the amount of data created during the detection phase, reducing the blocks required for accurate rendering and reducing render time, without affecting render quality.&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FcfyZmlPuWaQzBxDhYxJG%2FScreenshot%202026-01-07%20105102.png?alt=media&amp;token=58ca481d-69ce-4d8a-b0ca-a1a6505d1569" alt=""><figcaption></figcaption></figure>

[<br>](https://support.presenzvr.com/docs/html/_images/InsideZOV.png)**When to use the chaotic tag:**

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FjCXHImvJ2muactMVb63S%2F15.png?alt=media)

If you’re working on a scene with complex geometry, such as vegetation or hair, you can speed up your rendering process hugely by using the PresenZ Chaotic tag. This tag reduces the number of unnecessary points rendered for complex objects, decreasing the overall render time and the size of the final .prz file&#x20;

**How does it work?**

To achieve this, V-Nova PresenZ divides the space around the Zone of View into smaller volumes corresponding to 8x8 pixel patches, with a variable depth. \
As the distance from the center increases, the slices become deeper, as we don’t require as much detail further away from the ZOV.\
\
This is a top view representing the space partition:

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FGkoufqgyvLeCN4zPeb5d%2F16.jpeg?alt=media)

For each patch, V-Nova PresenZ computes a maximum of 8x8 pixels (64 pixels) instead of the original complexity of your objects. Without this process, leaves or grass could generate hundreds of points for the same patch volume. This approach can significantly reduce the complexity of your scene and speed up rendering.&#x20;

We provide a UI to tag as "chaotic" selected objects in your 3D scene.

## Atmospheric rendering

#### Smoke, fire and atmospheric rendering&#x20;

V-Nova PresenZ does support Smoke, Fire, atmospheric and more broadly "volumetric" rendering from Maya Arnold. But it is important to remember that a volume that takes up a large space will greatly increase the number of points and the amount of data in the scene. So this will take up more blocks in the render. As such, you should limit the amount of space used for atmospheric volumes by using the aiAtmosphereVolume shader on a geometry. As seen below

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FDZ7BQOym5RnDJRugaoDP%2Fimage.png?alt=media&amp;token=76a41979-f8d2-415a-ad5d-3316df538866" alt=""><figcaption></figcaption></figure>

You should avoid the atmosphere attribute inside Arnold, as this will create points throughout the scene and will affect your renders, as mentioned. It will also create issues where you will need to render inside the Zone Of View.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F4uTygkQaT5ZVvobEv6wr%2Fimage.png?alt=media&amp;token=9be63ae9-4799-4d68-9694-3724797558c1" alt=""><figcaption></figcaption></figure>

When rendering any volume, it is important to limit the number of effective Lights and improve the samples of the render inside of Arnold

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FYtaOLhZJLmHKbhufL0SK%2Fimage.png?alt=media&amp;token=1e47ed0f-d469-4cee-a9dc-f35f3c1e3e24" alt=""><figcaption></figcaption></figure>

* Here is a simple look at render settings that will work for testing

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FEcVC30oDHIniNR2n5qIz%2Fimage.png?alt=media&amp;token=b60c541a-d667-40cc-95a0-d1d81ef5e2e8" alt=""><figcaption></figcaption></figure>

## Clipping Sphere

{% embed url="<https://vimeo.com/1152245634>" %}

The clipping Sphere tool is a useful addition to PresenZ and allows artists to start learning the concept of layers. It allows us to render only what is visible inside the sphere, which will allow us to use are 3 part method of layering. We tend to use 3 main layers.

* Foreground ( animated ): this is any animated characters or the main animated feature of the scene
* Background ( static ), this is for environments that would be static but are close enough to require details to be rendered&#x20;
* 360-degree image (Video at infinity), this is the background sky dome or surrounding image

The process to get this working is very simple.

* Start by enabling the clipping sphere inside the PresenZNode

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FkvegPPgPnFbpHwpo46vu%2Fimage.png?alt=media&amp;token=52b08fb8-bbbd-47d9-b756-2c1ef1790436" alt=""><figcaption></figcaption></figure>

* Scale the sphere to contain the selected geo you want to separate, be careful to include the interaction with the environment, e.g., shadows

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FK2mV96Qutsa8teCgsG6B%2Fimage.png?alt=media&amp;token=afd2ee2f-be38-4419-9432-49138bb65d66" alt=""><figcaption></figcaption></figure>

* Use the "Mode" attribute to render both inside and outside separately.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FVx9zqi6VyqVqBZglFiNk%2Fimage.png?alt=media&amp;token=404cced8-2736-47e8-8921-2d91cd20448b" alt="" width="363"><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FKNO5n4EiVEBOvXuWD9o5%2Fimage.png?alt=media&amp;token=2a46a0c6-3089-497d-b42c-ea1113ba7e86" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fi9ZITVSEmW3VCwAEvlkO%2Fimage.png?alt=media&amp;token=e155430c-8aaa-4c75-b111-8dd87d7b1b99" alt="" width="375"><figcaption></figcaption></figure>

You can then use the player to layer the image to view it correctly, as shown below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FRO5mWQ86C8BguViTr6d0%2Fimage.png?alt=media&amp;token=c7c0a654-105e-4162-89c8-d3c7f68ab014" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FzhJ5Fgaa6Djs5OJqEB7t%2Fimage.png?alt=media&amp;token=2f1307ec-8687-47d3-a3fe-176c0a5b43ca" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Note:** while we advise the three layered setup for simplicity and for use in the V-Nova Presenz Player and Sequencer, this is not the only way to create layers and it can be useful to know that any regular layer setup can be used to create simplified layers and then have them merged in the comp stage. for more information on this you can check the Getting Started with V-Nova PresenZ Houdini.
{% endhint %}

&#x20;**The clipping sphere**

The clipping sphere feature in V-Nova PresenZ allows you to render only a specific part of a 3D scene. This can be useful if you’ve made changes to a particular object and want to see how those changes have affected it without having to render the entire scene. Instead, you can place a clipping sphere around the object and render only that part. This is similar to doing a cropped render in 2D.

In the rendered image, everything outside of the clipping sphere is not visible, giving the impression that the object is isolated in space. In some cases, you might want to do the opposite and render everything outside of the sphere, creating a hole in the final render.

This technique can also be used for animations where a large part of the scene is static and unaffected by the animation (such as shadows and reflections). By rendering this static part of the scene once using the clipping sphere feature, usually outside the sphere, you can save time and resources during the animation rendering process, by just rendering inside the sphere for the animation.

![](https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F3OOAWordhgNfQaUzas13%2F14.jpeg?alt=media)


# Getting Started with V-Nova PresenZ Houdini

This section is dedicated to getting you up and running with V-Nova PresenZ in Houdini

Getting started with Houdini and the compositing process can feel daunting, but this section is designed to make it straightforward.&#x20;

We’ll cover the basic export workflow, how to start manipulating the generated points, and best practices for compositing them into a VR-ready scene. To begin, here’s a brief overview of what you’ll need in Houdini and how to set it up.

{% hint style="info" icon="lightbulb" %}
**Note:** if you have skipped the Maya and Arnold sections please make sure that you understand each of the "Getting started" sections before proceeding to Houdini, as these are in a linear workflow simulating the production process and there is information in these sections that you will need in the following sections.
{% endhint %}

#### Basics

* [Exporting AOV's and Light Groups](#exporting-aovs-and-light-groups)
* [Getting started Houdini basics](#getting-started-houdini-basics)
* [Compositing with AOV's](#compositing-with-aovs)
* [Creating Masks](#creating-masks)
* [Using Cryptomattes](#using-cryptomattes)

## Exporting AOV's and light groups

{% embed url="<https://vimeo.com/1166724448/fbd06e7764?fe=ci&fl=sv&share=copy>" %}

If you’ve worked through the Maya and Arnold section, you should be familiar with the current setup. In the render settings, we’ve enabled AOVs such as **ID**, **Albedo**, and **Pref**, and we’ve also added light groups under **RGBA\_sky**, **\_lamp**, and **\_sun**. In PresenZ, this data is applied to the generated points so that, when you rebuild them in Houdini (or another compositing package), they behave as a coherent image layer, similar to what you’d expect from a standard, uncompressed image file (for example, an **.exr**).

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F721XK2NRxqtm8PbpDxuY%2Fimage.png?alt=media&amp;token=fb8fbcc7-7acb-4233-9f8e-48daeb1366e9" alt="" width="280"><figcaption></figcaption></figure>

It’s also worth noting that, in addition to adding AOVs, we can separate the scene into layers in Maya. I’ve linked the Maya Clipping Sphere section [<mark style="color:$primary;">here</mark>](/technologies/v-nova.presenz/getting-started/getting-started-with-v-nova-presenz-maya-and-arnold#clipping-sphere); we can use that approach, or Maya’s render layer exports as shown below. The goal is the same: avoid re-rendering multiple frames of a static background, and render animated characters separately to reduce render time and cost.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FbzA4pRJu8k56EKjDcxD9%2Fimage.png?alt=media&amp;token=7b236af9-48f6-4481-bb86-dffed3465873" alt="" width="362"><figcaption></figcaption></figure>

{% hint style="info" icon="lightbulb" %}
**Note:**  If you’re rendering animation, make sure you have Motion Vectors setup, as these values are also applied to the points generated and will be vital for smooth interpolation on animation.
{% endhint %}

## Getting started Houdini basics

Now lets move on to the more daunting part, creating a PresenZ setup is remarkable easy. This section will go over the step by step of creating a Houdini PresenZ scene and outputting a .PRZ file so that you can view in the player or sequencer.

{% embed url="<https://vimeo.com/1166717390/943e9bf8be?fe=ci&fl=sv&share=copy>" %}

Lets start by creating a PresenZ node in the object level, this will be our main workspace going forward and is simply a modified Geometry SOP node.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FOpJ6Du2303C2yTDfTAnZ%2Fimage.png?alt=media&amp;token=1868e890-7488-4d66-81c2-3e98b1622496" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F90RSue364ovG4zDV6afT%2Fimage.png?alt=media&amp;token=4f12cfef-d957-415c-a656-b5377d960523" alt=""><figcaption></figcaption></figure>

Inside of this node you will see that there is a basic setup ready. If you have a .przRender file created from maya this is where we will add it in the Loader.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F3LJEE3xxluKD58gEGt2b%2Fimage.png?alt=media&amp;token=72f50762-ee2d-478f-a844-fb3ae230012d" alt=""><figcaption></figcaption></figure>

To manipulate PresenZ data you always need to start with the Loader PresenZ node.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FVrAWOkaab1NZyx7RN5vO%2Fimage.png?alt=media&amp;token=850bbeca-0225-48e9-896f-ed31efd23fdc" alt=""><figcaption></figcaption></figure>

Its important to note that here in my example I have added both left and right eyes, but the process is the same. Simply use the Load PresenZ node to select the files required,we then use the copy\_right\_to\_left\_eye node. This is important as it will append the AOV's and values with correct left and right eye values, which we will see in the spreadsheet.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FAckWOk7nSMDA4HXUzqTw%2Fimage.png?alt=media&amp;token=c9c7e536-6741-4949-8851-5fb330b6a46f" alt=""><figcaption></figcaption></figure>

If you have, like me, separated parts of the scene using the clipping tool, you will have multiple layers to call in as seen below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FAf8v2CruWJDjHU2W9nFy%2Fimage.png?alt=media&amp;token=6389ba0f-900e-437e-bbd2-8bcf0348b49e" alt=""><figcaption></figcaption></figure>

It’s worth noting that the multiple blocks shown above are expected. For example, Outside Left and Outside Right each contain three blocks (0–2). This comes from the render blocks explained in the Maya Arnold section ([<mark style="color:$primary;">link here</mark>](/technologies/v-nova.presenz/v-nova-presenz-documentation/render-blocks-and-occlusion-factor)). It’s important to label these correctly at this stage to avoid confusion later.

As you can see, you can use a simple **Merge** node to combine the left-eye blocks and, separately, the right-eye blocks. However, when it comes to combining the left and right eyes, use the **copy\_right\_to\_left\_eye** node again. This ensures all the required information from the right eye is copied across and that the attributes are named correctly for the steps that follow.

In short, merge the blocks per eye with a simple Merge node, then use **copy\_right\_to\_left\_eye** to combine the left and right eye renders with the correct data and attribute naming.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FNdXt5kItLxscU5W84E8N%2Fimage.png?alt=media&amp;token=2b86b65f-2062-43e3-b281-75a7ffa73239" alt=""><figcaption></figcaption></figure>

If you wish to create new Loader nodes, simply tab and type Load and you should see the node available, and if you need to create a new write node it is the same bellow.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FpIhKy6ovHel0Zxx827yM%2Fimage.png?alt=media&amp;token=62611758-5101-42cb-8dc1-1e5bfeebdc40" alt=""><figcaption></figcaption></figure>

Now lets setup the write node. We just need to tick the "save PRZ file" control and set the file path for the output, the rest of the setting should be set from the scene, so you should check the start and end frames.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F1JBnWBSVUmn2fPgyV1uG%2FScreenshot%202026-01-22%20173547.png?alt=media&amp;token=361898ed-87b5-4334-a42f-90f564ade244" alt=""><figcaption></figcaption></figure>

And we are now ready to render for V-Nova PresenZ Player or Sequencer. Once done, simply locate your .prz file and either double click if you have the dashboard tools installed and setup or use the player and drag and drop your .prz file into the work space. You can then view in VR or Monitor view depending on preference.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FBprJll14fD16dJPHdLV3%2Fimage.png?alt=media&amp;token=67524ab7-2ac9-4cf5-9773-030673efc1e9" alt=""><figcaption></figcaption></figure>

### Load PresenZ node (prz\_loader)

The **Load PresenZ** node, identified as *prz\_loader*, is a geometry node in Houdini designed for importing and converting PresenZ files (przDetect, przRender, or prz) into Houdini point clouds. This node is crucial for volumetric renders that leverage point cloud data.

### Input file formats

This node accepts three types of PresenZ files:

1. przDetect
2. przRender
3. prz

Each file type is processed and converted into a Houdini-compatible point cloud, retaining essential metadata in the detail attributes.

### Left eye branch

* **Load PresenZ Nodes**: Two nodes named *THEROOM\_Tutorial\_L\_bloc0* and *THEROOM\_Tutorial\_L\_bloc1* are used to import the left eye’s point cloud data.
* **Merge Node**: A merge node labeled *merge6* consolidates the point cloud data from the two Load PresenZ nodes.

### Right eye branch

* **Load PresenZ Nodes**: Similar to the left eye branch, two nodes named *THEROOM\_Tutorial\_R\_bloc0* and *THEROOM\_Tutorial\_R\_bloc1* load the right eye’s point cloud data.
* **Merge Node**: Another merge node labeled *merge7* combines the point cloud data from the right eye’s Load PresenZ nodes.

### Combining left and right eye data

* **PresenZ Combine Node**: The *prz\_combine\_LR* node takes the merged output from both the left and right eye branches. It is specifically designed to handle stereoscopic data, ensuring the point clouds are combined in a way that preserves the spatial and visual integrity of the stereoscopic effect.

#### Write PresenZ node (prz\_writer)

The **Write PresenZ** node, referred to as *prz\_writer*, is a geometry node in Houdini used for exporting point cloud data into PresenZ file formats. This node is pivotal for creating przRender files or prz files from within Houdini.

#### Supported output formats

The **Write PresenZ** node supports two primary output formats:

1. **przRender Files**: Used for rendering purposes, carrying detailed color and spatial data.
2. **prz Files**: Typically used for storing compressed point cloud data.

#### Color space consideration

An essential aspect of using the *prz\_writer* node is the color space of the point cloud data:

* The node assumes that the color data of the point cloud is in a **Linear/ACEScg Color Space**. This assumption is critical for ensuring that the output files maintain color accuracy and consistency with the intended design.

## &#x20;Compositing with AOV's

{% embed url="<https://vimeo.com/1166712224?fe=ci&fl=sv&share=copy>" %}

Now let’s go over the process for using the AOV's in order to edit Colour in our Houdini scene and improve our render. So what are AOV's?

So what are AOV's?

Arbitrary Output Variables (AOV's) are render passes that break down a 3D scene into specific data layers (like reflections, shadows, depth, normals, material IDs) beyond just the final colour ("beauty pass"), giving compositors control to adjust elements individually in post-production, enabling more flexible adjustments (lighting, colour correction, motion blur) without re-rendering the entire scene.

In PresenZ, we store all of this data on the points that we generate. This allows us to do very similar adjustments to what you would see in any post-production workflow. If you are not used to comping, this process might seem odd, but it simply allows the artist to tweak and adjust the final render without re-rendering, allowing us to beauty light characters like we see in post film production.

For comp artists, this process will be nothing new, but Houdini might be new software that we don't usually associate with compositing. Let me be clear: the process is essentially the same. If you are used to Nuke or other compositing packages, you will find Houdini operates much the same way. Instead of using a 2D image, we have 3D points. Nuke has been able to visualise points since version 6.0 in 2010, so most Nuke artists will be aware that points can be used to hold values for compositing. Houdini just allows us to use points more easily.

So that aside, let’s start.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FIcbqaWVLRbzXia4AmvPX%2Fimage.png?alt=media&amp;token=1d52977e-cff7-48e3-b18f-5be946beb2bc" alt=""><figcaption></figcaption></figure>

Taking a look at the image above, we can see a list of available AOV's that have come through on the points. These are on each of the Load PresenZ nodes and will need to be activated, but there is a handy **Activate All** at the bottom of the list.

Now we need to add a Unpremult. For those artists who have used comp packages like nuke this is nothing new. Unpremulting an image before working in compositing is a critical step because it allows you to colour correct, grade, or manipulate the colour values of a masked object without affecting its transparency or creating dark, unwanted halos around the edges. Dividing the RGB by the alpha restores the original colours of the image. The grade is applied to the image data, not the boundary pixels, resulting in clean, natural, and accurate colours.

we can find the **unpremult** and **Premult** nodes by tabbing and typing in the search.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F9zMWPC4pCp6B0ss9mPaS%2Fimage.png?alt=media&amp;token=a67c9181-9919-4331-8881-fc8d4014e0e8" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FGTzwQIfG4eWaLgCFZrCs%2FScreenshot%202026-02-25%20145251.png?alt=media&amp;token=1c359a2e-21ec-4e6b-af2c-e57a4ca6e3f3" alt=""><figcaption></figcaption></figure>

From here, we should move to the compositing node. This can be found by simply hitting **Tab** and typing **comp**, and it should appear in the dropdown menu, as shown bellow. Now double-click to move inside.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F7KcrffEMfLvXDqheU74F%2Fimage.png?alt=media&amp;token=6f2ab037-4a8c-4d41-ac23-c9815dcfe969" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fe5vlUvaWZKl2xdBgNme8%2Fimage.png?alt=media&amp;token=080c93b3-8b75-4acc-84fe-1dce6d897435" alt=""><figcaption></figcaption></figure>

Inside the comp node, we can see we have a simple setup ready for us. We can also see there is a brief description/notes to show us how this functions. We can ignore the input 2 section for now, and let’s look at a very simple adjustment of our light groups.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F7LyjGGwvTUHNELF4oOV5%2Fimage.png?alt=media&amp;token=84e8ab9a-e15e-42bc-91c1-fea4657ef407" alt=""><figcaption></figcaption></figure>

Let’s start by creating the basic adjustment nodes: an attribvop with an attribadjustcolor node, and then an attribute combine to merge with the original. We can go over the settings one by one, but let’s start by double-clicking the attribute VOP to go inside.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FXOdK53WBcfjOkru5q7RT%2Fimage.png?alt=media&amp;token=a8c25ef4-a692-4ec2-9579-e9b5385dc9b9" alt=""><figcaption></figcaption></figure>

Inside, you should be greeted with the image above. We won’t need the Geometry VOP Global. We need to add a parameter node to identify the attribute to edit, so let’s start with a light group and create an ADD node to adjust it before plugging into the **Cd** value on the **Geometryvopoutputt**, shown below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FfgmsyQks6LfV7pLsGTlO%2Fimage.png?alt=media&amp;token=a348d2da-663d-424f-86dc-11c600072b34" alt=""><figcaption></figcaption></figure>

Below is an example of hooking up the **RGBA\_sky** AOV into the ADD and **Geometryvopoutputt**. Here, I added it as a **Color**, as my values are RGB, not RGBA, but we can change the type to **Color+Alpha** if you have just RGBA values. Note that this was a preference, but RGB is better for use in PresenZ.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FadACq3EaMBaSCiT90W1q%2Fimage.png?alt=media&amp;token=2002d369-9673-4e31-a52b-cdd7fc7f38e2" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F6g6OPL7Mz6Vj1RqYPxvZ%2FScreenshot%202026-02-25%20145657.png?alt=media&amp;token=254f79a6-d5cd-41a4-847b-d7385aab06cd" alt=""><figcaption></figcaption></figure>

you can also get a list of available AOV's from the **Geomatry Spreadsheet** seen above&#x20;

Having added all the lights, you can see the basic setup below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FmJqQtmD1R2WoHlkHOtG9%2FScreenshot%202026-01-26%20112949.png?alt=media&amp;token=819424ae-1aee-42da-ac44-ee933b0b3069" alt=""><figcaption></figcaption></figure>

Below, you can see the difference in the render when we add a bit more warmth to our lamps and a bit cooler blue to the sky. by adjusting the **Constant color** and changing the **Operation** to a **multiplier .** The bottom image is with the **attributadjustcolor** node disabled.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FjG7FY77JZJ28Bs4oebMX%2Fimage.png?alt=media&amp;token=26b540ee-72bd-4c3e-a249-fd7f3220ba12" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F8WLhDA3pTMcpcLHGYRWT%2Fimage.png?alt=media&amp;token=66adf649-9f27-4aea-959e-b30f75540e87" alt=""><figcaption></figcaption></figure>

This shows how we can quickly and easily adjust the lighting in our Houdini comp node, but perhaps you will need to adjust these lights on a specific object, so now let’s look at how to isolate objects and apply similar adjustments.

#### Color management

The point cloud colour data is in a linear/ACEScg color space. In order to preview the color properly in the Houdini viewport, please make sure that the setup does a proper linear to sRGB correction.

The color correction panel can be found here. Please consult the Houdini documentation for OCIO management. By default, Houdini will use a gamma 2.2 correction that is mostly correct.

<figure><img src="https://support.presenzvr.com/docs/html/_images/viewport_CC.png" alt="PresenZ Node Network Diagram"><figcaption></figcaption></figure>

<figure><img src="https://support.presenzvr.com/docs/html/_images/viewport_CC_2.png" alt="PresenZ Node Network Diagram"><figcaption></figcaption></figure>

#### Compositing with przRender files

For compositing purposes, it is recommended to use przRender files. These files contain data in floating-point format and are in a linear space/ACEScg color space. This ensures high-quality rendering outputs, suitable for professional-grade compositing.

#### Point cloud attributes

The point cloud generated by the **Load PresenZ** node has several modifiable and non-modifiable attributes:

#### Modifiable attributes:

* *v\@Cd*: Color of the point. The color for the right eye is encoded in *v\@Cd\_Right*.
* *f\@Alpha*: Alpha value of the point. Modification is discouraged unless necessary.
* *v\@v*: Motion vector information in 3D space. It defines the position of a point in the next frame using the formula *v\@P = v\@P + v\@v* in an attribute wrangle node.
* *v\@N*: Normal of the underlying plaque defined by the current point. Modification is discouraged without a valid reason.
* You will also find your AOVs as others point attributes.

#### Non-modifiable attributes:

* Other values like *v\@P*, *i\@camera*, *i\@posx*, *i\@posy*, and *f\@distance* are crucial for saving the point back into the V-Nova PresenZ format. Modifying these may result in incorrect prz files or be ignored entirely.

#### Manipulation of points:

* Points can be removed, merged, grouped, or isolated as needed for compositing purposes.

#### Detail attributes

The point cloud’s detail attributes provide vital information about the render settings used:

* *draft*: Indicates the draft mode setting.
* *eye*: Specifies which eye (left or right) the render corresponds to, important in stereo rendering.
* *renderscale*: Provides the scale factor used in the render.
* *AOV names*: Lists the names of the Arbitrary Output Variables involved in the rendering process.
* *box scale*: Indicates the scale of the Zone of View used in the render.
* *imageSizeX* and *imageSizeY*: Provide the dimensions of the rendered image in pixels.

These detail attributes are essential for understanding the context and settings under which the original file was rendered, and may influence how the point cloud is manipulated or re-rendered.

### Creating masks

{% embed url="<https://vimeo.com/1166710516/8cebeaf861?fe=ci&fl=sv&share=copy>" %}

In compositing, a **mask** is a tool that controls the visibility or effect of an image or video layer, acting like a stencil to reveal (white), hide (black), or partially show specific areas, allowing for precise blending, isolation, or selective application of effects without permanently altering the original pixels.&#x20;

When creating a mask, we use the PresenZ Mask node. You can create a Mask node by hitting **Tab** and typing **Mask** while in the node graph inside our PresenZ Geo.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FPHv007yYoZx28oj5aiBk%2Fimage.png?alt=media&amp;token=05fb9d47-66bc-4d22-9157-10032f72004c" alt=""><figcaption></figcaption></figure>

We don’t need the second input, which is for tracking data, but in this example we will isolate our ball and use the mask so that we can confine our Color Correct to it.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FOY90lnC5qo32M2YFZ1h9%2Fimage.png?alt=media&amp;token=5c5e3c9e-9e96-4ef0-b945-5c1a49f71c8f" alt=""><figcaption></figcaption></figure>

We can see we have the ball isolated by the fact it is shaded white.&#x20;

Here, we are going to use the Color Correct to boost the red of the ball, but we could just as easily change the colour completely.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FkRABhzRZACVHG8WqfKJy%2Fimage.png?alt=media&amp;token=930eb2d2-d7f3-4cf9-aeff-6a9220f02160" alt=""><figcaption></figcaption></figure>

This method can be useful for separated objects, but it has issues when objects are interacting with others or are moving. In this case, you can place the mask object carefully, but cryptomattes are easier, as we will explain.

#### PresenZ background mask node (prz\_background\_mask)

The PresenZ Background Mask node, named **prz\_background\_mask**, is used within a Houdini network to generate a background mask attribute for scenes using **przRender** files. This mask identifies regions of the render that are not occluded by any geometry, effectively marking the background.

<figure><img src="https://support.presenzvr.com/docs/html/_images/background_mask.png" alt="PresenZ Background Mask Node Network"><figcaption></figcaption></figure>

#### Functionality

* **Input:** This node processes data from **prz\_loader** nodes that contain **przRender** files. For the mask creation process, only the loaders for the left eye are necessary.
* **Background mask creation:** The **prz\_background\_mask** node adds a **“background mask”** detail attribute to the point cloud data. This attribute distinguishes between foreground (occluded by points) and background (unoccluded regions).
* **Baking the mask:** After the **“background mask”** detail attribute is created, it can be baked back into the network, allowing for further processing or output.

#### Attribute copying

An attribute copy node, named **attribcopy1**, is used to transfer the **“background mask”** detail attribute to the relevant points in the point cloud. This ensures that the attribute is consistently applied throughout the data, preparing it for the final baking process.

#### Final output

The output of this network is an image baked with a background mask inside the **.prz** file. This can be used in subsequent compositing or rendering processes to isolate or replace the background with alternate imagery or effects.

#### Usage instructions

1. Connect the **prz\_loader** nodes for the left eye to the **prz\_background\_mask** node.
2. Ensure that the **“background mask”** detail attribute is correctly configured in the **prz\_background\_mask** node.
3. Use the attribute copy node to apply the **“background mask”** attribute to the entire dataset.
4. Connect the output to the **prz\_writer** node to bake the mask into the final image or to proceed with further adjustments.

#### Note

This node is particularly useful in virtual production and compositing workflows where background elements need to be selectively adjusted or replaced.

### Using Cryptomattes

{% embed url="<https://vimeo.com/1166723214/dd62bcacad?fe=ci&fl=sv&share=copy>" %}

In compositing, a Cryptomatte is a powerful system that generates automatic, pixel-perfect ID mattes (masks) for objects and materials directly from your 3D renders, allowing you to easily select and manipulate specific elements in post-production without re-rendering, even with motion blur, transparency, and depth of field. It works by encoding object/material IDs into special image layers, which compositing software can then interpret to create masks by simply selecting an element in the render.

Let’s start by applying a Cryptomatte as a mask. We can use a Cryptomatte by creating an **Apply Cryptomatte** node inside our PresenZ workspace, by hitting **Tab** in the node graph and typing **Crypto**, as shown in the image below.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F1kIKqzniDxobUXwiC2fq%2Fimage.png?alt=media&amp;token=178abd3d-4862-4d61-b0f7-f4e373505c05" alt="" width="499"><figcaption></figcaption></figure>

This then allows us to isolate a Cryptomatte to use as a mask. Simply use the cog and select your asset. In this case, we will isolate the **ballShape**.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FCf8EZrbPR6Ow6qRCgu61%2Fimage.png?alt=media&amp;token=32902a7f-b919-42e6-be58-d7e10662f1f7" alt=""><figcaption></figcaption></figure>

You can also find a list of cryptomattes in the **Geomatry\_spreadsheet** under **cryptomanifest**&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FVqA3LaS9NgMTN7sgdmTy%2Fimage.png?alt=media&amp;token=97a56ee5-ba0d-46cb-8e7b-296f9ab55204" alt=""><figcaption></figcaption></figure>

Now all you need to do is use this mask/Cryptomatte to edit the scene. In this case, I am editing the colour of our ball. Now I want it purple.&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FBQauaCCndaUaZBEOd5nT%2FScreenshot%202026-02-25%20145905.png?alt=media&amp;token=91b13b38-68c9-4ffc-b56d-a3f2d5f7ec78" alt=""><figcaption></figcaption></figure>

Now all you need to do is use this mask/Cryptomatte to edit the scene. In this case, I am editing the colour of our ball. Now I want it purple.&#x20;

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fi3u2LfjLsDITa7TcCFJN%2Fimage.png?alt=media&amp;token=95135070-3446-4a8b-a281-30bae5e70987" alt=""><figcaption></figcaption></figure>

We can use this method to update specific objects or assets in the scene without affecting anything else. It doesn’t require tracking data or animation like a mask, as it is a pass in the render coming from the object.

#### PresenZ Apply Cryptomatte

The **Apply Cryptomatte** HDA node is used within a Houdini network to generate a mask attribute using Cryptomatte AOVs.

<figure><img src="https://support.presenzvr.com/docs/html/_images/cryptomatte11.png" alt="PresenZ Cryptomatte Mask"><figcaption></figcaption></figure>

### Parameters

* **Type:** Specifies the type of Cryptomatte (object, material, or asset).
* **Mask Name:** Defines the group name where the points corresponding to the hashes will be added.
* **Hashes:** A list of hashes to search for. These hashes can be found in the corresponding **.json** files located next to the **przRender** files.

### Notes

* There are a series of corresponding AOVs named **“\_mask0”** and **“\_mask1”** that contain the percentage of ownership for the hash. A value of **1** indicates exclusive ownership, meaning no other hashes correspond to that point.
* These AOVs should be used in cases where colour correction is applied, allowing you to multiply the effect of the correction by the ownership value. Unfortunately, Houdini groups do not support weighting, so implementing this feature is left to the user’s discretion.

## PresenZ visualizer node (prz\_visualizer)

The PresenZ Visualizer node, **prz\_visualizer**, is a geometry node in Houdini that transforms point cloud data into geometry representations. This conversion aims to closely mimic the output that will be displayed by the PresenZ Player, providing a more realistic preview of the final render.

### Functionality

The PresenZ Visualizer node serves as a visualisation tool, converting point cloud data into geometry. This is particularly useful for previewing how the final render will appear in the PresenZ Player.

### Usage considerations

* **Editing:** The output from the PresenZ Visualizer node is not intended for editing. Any modifications should be made prior to this visualisation stage.
* **Shading effects:** Shading applied to the geometry can alter the perceived colour of the scene. This may result in a visual difference between the visualiser output and the original point cloud data.

This node is useful for a more accurate preview during development, but users should be aware of its limitations around editing and colour fidelity.

That will about cover the basics, in the advanced section we will try to show how we use all of these nodes in unisan.  Showing how we can take a basic and simple setup into a fully working production model for a more complex project.


# Getting Started with V-Nova PresenZ Player

Use V-Nova PresenZ Player to build and view PresenZ shots and sequences in VR. The player supports basic finishing adjustments and performance monitoring. Use its overlays and graphs to diagnose playback and rendering issues.

{% embed url="<https://vimeo.com/1167393933?fe=ci&fl=sv&share=copy>" %}

{% embed url="<https://vimeo.com/1167394284?fe=ci&fl=sv&share=copy>" %}

### Introduction

V-Nova PresenZ Player is a compact application for building and viewing PresenZ-based VR shots. It provides utilities for working with PresenZ files. It also includes performance graphs for diagnosing issues and improving playback.

**Features**

* View PresenZ content in VR
* Manage files with drag and drop
* Monitor performance with visual graphs
* Apply basic post-processing adjustments
* Customise a focused user interface

### UI overview

V-Nova PresenZ Player has a focused interface. Access its actions from the **Movie**, **View**, and **Performance** menus.

<div align="center"><figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fh9eaUi8pIm8gZwiZTvWG%2Fimage.png?alt=media&amp;token=4481c3fd-bfd8-4dd9-a97c-d3ca6be9293a" alt="" width="375"><figcaption></figcaption></figure></div>

<div align="center"><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FE6WRs2sNxsLZil3RmftT%2Funknown.png?alt=media&amp;token=823d4544-25c3-4daf-991f-a5f839729e2a" alt=""></div>

{% tabs %}
{% tab title="Movie" %}

* Open
* Movie info
* Scene info
* Frame info
  {% endtab %}

{% tab title="View" %}

* Keys
* Playback
* Log
* Display colours
* About
* Inspect particles
* Adjust scene
* Postprocessing
* Shader tweaks
* Camera position
  {% endtab %}

{% tab title="Performance" %}

* Frame loader
* Audio
* Rendering
* Frame graph
* GPU memory
  {% endtab %}
  {% endtabs %}

### Supported media

The primary media type is <mark style="color:blue;">.prz</mark>. This losslessly compressed format is generated from <mark style="color:blue;">.przRender</mark> files. It supports efficient playback in the player.\
\
The player supports these file types:

#### PresenZ formats

* <mark style="color:blue;">.przShot</mark>
* <mark style="color:blue;">.prz</mark>

#### Background images

* <mark style="color:blue;">.png</mark>
* <mark style="color:blue;">.jpeg</mark>
* <mark style="color:blue;">.pitex</mark>
* <mark style="color:blue;">.exr</mark>

#### Background video

* <mark style="color:blue;">.mp4</mark> (HEVC)

#### Audio

* <mark style="color:blue;">.opus</mark>

## Movie

### Open and load media

Load media by dragging files into the workspace. Alternatively, select **Movie** → **Open**. The file browser lets you add files or open a <mark style="color:blue;">.przToc</mark> file for editing.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F5YDU4o2xX1sV6gkOHadN%2Funknown.png?alt=media&amp;token=e3e93c4e-6193-48b0-9a9b-de548b5055d7" alt=""><figcaption></figcaption></figure>

### Builder

Use the Builder to create and test shots. It automatically adds files dropped into the workspace. The Builder saves the scene as a <mark style="color:blue;">.przToc</mark> file. See [File formats and structures in V-Nova PresenZ](broken://spaces/i6eg1dFhczcq6mxQYODq/pages/o95AgbmklULmM7R1WCsv) for file-type details. The Builder header shows the file location and common directory.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FR37HdQJpwuDs21ZEVSWz%2Funknown.png?alt=media&amp;token=169d3d4c-ddbd-42f3-9d56-f2fef8016627" alt=""><figcaption></figcaption></figure>

Added files appear in the **Files** section. Assemble the shot in the **Scene** section. This layout reflects the PresenZ three-stage construction. Drag files into the appropriate section.

**FG**: (Foreground) The Animated PresenZ Volumetric layer\
**BG:** (Background) The static PresenZ volumetric layer\
**Env**: The 360 environment sphere image or video layer<br>

### Movie info

View the number of frames and scenes, the current position, and file locations. Use this window to quickly identify sequence and frame numbers when investigating artifacts.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fr6vWpwEmAzhuAVT7T0rQ%2Funknown.png?alt=media&amp;token=9e3af0c0-2736-4a99-b01c-3f0a306006cf" alt=""><figcaption></figcaption></figure>

### Scene info

View information about the current scene. Create, remove, and switch scenes in the Builder. You can also use keyboard shortcuts.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FJyHRyFR8nJKNMASahqZH%2Funknown.png?alt=media&amp;token=876edb16-d159-41e0-b11b-07e41b031609" alt=""><figcaption></figcaption></figure>

### Frame info

View the current layers, particle counts, motion vectors, compression, and format.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FLgRnochTcWJuKzflbaCo%2Funknown.png?alt=media&amp;token=f0751787-ac35-4a9e-8ae4-1f03f3b3dad3" alt=""><figcaption></figcaption></figure>

## View

### Keys

View the current keyboard shortcuts.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FfzYvtt0DMm6ApMPYMtAJ%2Funknown.png?alt=media&amp;token=63874ceb-f74b-46b5-a142-72c8d9d6146c" alt=""><figcaption></figcaption></figure>

### Playback

Control shot playback speed and switch scenes.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FTqKaaLTaVlY08xn3XL3y%2Funknown.png?alt=media&amp;token=ccc9dc33-d81a-4d14-94d4-a067a7d52cd1" alt=""><figcaption></figcaption></figure>

### Log

View information about actions in the scene. Some tools send diagnostic data here.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FJSlz7nPZET5JNL16mXNn%2Funknown.png?alt=media&amp;token=8d90c273-9c2e-4760-8bc2-b5ed0e2e32f9" alt=""><figcaption></figcaption></figure>

### Display colours

View and change colour settings to match the input material.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F7Tg9safA38UNEvRb3wtF%2Funknown.png?alt=media&amp;token=c9fbf9c9-e4ca-44b9-bf32-5d62060491ac" alt=""><figcaption></figcaption></figure>

### About

View the installed V-Nova PresenZ Player version.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FUEug4BlRk9cbSYdwJCPT%2Funknown.png?alt=media&amp;token=aef9fcbf-da55-47bf-9028-5ebbff4803f0" alt=""><figcaption></figcaption></figure>

### Inspect particles

Use this window with motion vectors to inspect particles and asset density. Enable the debug sky to replace the background with bright purple. This helps confirm that the PresenZ render contains complete geometry.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F7YkSBpWCqUXaGJwbwcyJ%2Funknown.png?alt=media&amp;token=f02ebf73-02f6-4318-98b8-a852f33b427d" alt=""><figcaption></figcaption></figure>

### Adjust scene

Make scene-level adjustments that persist until you reopen the movie. Rotate a background HDRI and adjust sound positioning. You can also set the viewing perspective before playback. This lets you position and rotate the VR agent away from the origin.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FMBWioj2XLTIXqnTuYyM2%2Funknown.png?alt=media&amp;token=78cec230-3fc9-4113-a9b5-dc1000eae076" alt=""><figcaption></figcaption></figure>

### Post-processing

Apply minor effects and adjustments to every scene. Available controls include fog, glow, brightness, and depth of field. These settings persist until you reopen the player.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FUtuGM9Q9VdgDuR8q0fPR%2Funknown.png?alt=media&amp;token=dda20734-784a-41da-9fd9-6dfaa008c279" alt=""><figcaption></figcaption></figure>

### Shader tweaks

This developer feature supports adjustments to custom shaders in the player. It is covered in the advanced documentation and is not intended for standard use.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FUfMg6CyVmZpIdJ527C0d%2Funknown.png?alt=media&amp;token=e5b681db-fa80-4cac-9482-cacd528bc946" alt=""><figcaption></figcaption></figure>

### Camera position

Copy values between cameras. This helps maintain a consistent viewpoint between shots and avoids jarring cuts. Use it to check the current camera position across multiple shots.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F1U7W5nS2nBZnGX0a9fmv%2Funknown.png?alt=media&amp;token=3930ee56-ee71-463c-8438-a74538c5007b" alt=""><figcaption></figcaption></figure>

## Performance

### Frame loader

View the frames currently loaded in the buffer and the playback frame rate. If the top bars reach the warning lines, the decoded-frame buffer may decrease.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F9jiY6jQbdLQbNgB7g0a1%2Funknown.png?alt=media&amp;token=5a69c4b2-ef22-4f7c-8c15-877d20f8d863" alt=""><figcaption></figcaption></figure>

### Audio

View the current audio and its load time. Use this information to diagnose sound issues.

<div align="center"><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Fwy4mewHxffWaDa7AXpvi%2Funknown.png?alt=media&amp;token=e589213d-5f79-45a8-b1d2-ef30bf735af8" alt=""> <img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FbZQpEHQOSX0JEBeKwQKo%2Funknown.png?alt=media&amp;token=e938c603-1162-4fc9-92e9-643b3578a090" alt=""></div>

### Rendering

View the current render speed and active processing. Use the graph to analyse dropped frames and slow playback.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2F0lI5m5BpH2n7gqC6Epc6%2Funknown.png?alt=media&amp;token=06a64395-334c-4fc4-a14d-503c4d4820fb" alt=""><figcaption></figcaption></figure>

<div align="center"><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2Ftpi2zhOCfxnYW3uWumKa%2Funknown.png?alt=media&amp;token=f859dc7c-6582-4636-a0a1-31547ae25854" alt=""></div>

### Frame Graph

View the high, low, and average playback frame rates. You can also send this information to the log. The window shows the number of accumulated frames at the top. Enable **Report to log**, then use **Pause** to capture data between log entries.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FScXrF0zQpuvVcLjUtBKD%2Funknown.png?alt=media&amp;token=fd89f9b9-5075-4568-84c6-7eb645c9afcd" alt=""><figcaption></figcaption></figure>

### GPU Memory

View current GPU memory usage, broken down by section.

<figure><img src="https://2927645607-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1djoRwqtVhAs0LOpvzC9%2Fuploads%2FbLwcjnX7aAZSHvN5JiDI%2Funknown.png?alt=media&amp;token=8f57d354-428d-4994-b5e8-7f3fa6ba4aa9" alt=""><figcaption></figcaption></figure>


# Introduction

The [V-Nova Platform](https://platform.v-nova.com) is a Software-as-a-Service platform for encoding MPEG-5 LCEVC-enhanced VOD and Live streams using the x264 and x265 base video codecs. The platform provides a way to evaluate the uplift available in compression and transcoding speed efficiency by applying V-Nova LCEVC to your content.

Before you start, please note the following:

* To play back LCEVC-enhanced streams, you must use an application with an LCEVC-enabled player. Players that have not been LCEVC-enabled will only play the lower-resolution base stream.
* Either log in at [player.v-nova.com](https://player.v-nova.com/), or download the demo apps from the Android, iOS and Windows app stores. Signing into these apps using your login credentials for the V-Nova platform will also enable you to view any encodes you complete on the platform.

&#x20;For additional information, please contact <platform@v-nova.com>.


# The dashboard

The V-Nova platform dashboard provides information about the platform and the current status of your account.

## **User dashboard**

The V-Nova platform dashboard displays information about the assets present in the account, the number of users, and depending upon the account type, information on remaining hours of credit.

![Platform dashboard](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/gcQmae9thNtr9cR78AqS/Trial_dashboard.PNG)

The following information is displayed in the platform dashboard:

* **Total VOD (Video on Demand) assets:** The **VOD** tile displays the total number of VOD assets present under this account. Clicking the tab navigates to **Encode jobs --> VOD** and lists the VOD assets.
* **Users:** The **Users** tile displays the number of users present under this account. For trial users, the **Users** tile always displays "**1**" since additional users cannot be added to this account.


# VOD Job list

## Displaying the Jobs list

To display the list of the jobs:

* On the left-hand navigation bar click the **Encode jobs** item and then click on **VOD**. The J**ob list** is displayed.

![VOD Jobs List](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/mp6xJCOLvOfH5ufSSz60/vod_dashboard.png)

The following information is displayed for each job:

* **Job ID:** Click the "**v**" icon to sort the Job IDs in ascending or descending order.
* **Job names:** User-speciified Job Name.
* **Manifest URL:** Auto-assigned **HLS (HTTP Live Streaming)** or **MPEG DASH Manifest URL** that enables the stream to be played in any compatible player. To copy the URL:
  * Highlight the text and right click copy.&#x20;
    * To exit the URL information display, click **OK**.

![Manifest URL Information](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/uS8bSKQBm0dcc9ltzW59/Manifest_URL_dashboard.PNG)

* **Duration:** Source content duration.
* **Catalogue Status:** **Enabled/Disabled**.
* **Status:** Encode status: **Encode Successful/Encode Unsuccessful**.

## Filtering the Jobs List

#### About Job filtering

Assets can be searched and filtered, using the filtering tools:

![Job List filtering tools](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/LUcSQX6v0dySz2Hlid4c/Filter_dashboard.PNG)

#### Searching for a Job

To search for a Job:

* Enter a full or partial name of job in the **Job name** field. Any results are displayed.

#### Setting and performing a Jobs list filter

To set a filter:

Click the **Open filter** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/SA44JOZA3yWVmI9OWGRc/expand_button.png" alt="" data-size="line">.  The **Job library list** dialog is displayed:

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/t6OgbEGlFwcR6RPt4dWp/job_filter.PNG)

The following fields can be specified as filters:

* **Job name:** Name of VOD asset.
* **Job ID:** Job ID of the VOD asset.
* **Type of packaging:**
  * **Byterange:** Defined by a range of bytes
  * **TS Chunks:** Transport Stream Chunks
  * **Webm**: Media file
* **Content ID:** Content ID of the VOD asset.
* **Date of creation:** Date of the creation of the VOD asset.

To perform a filter operation:

* Click the **Filter** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/RvKTe1az3CFpnLSVnMur/filter_button.png" alt="" data-size="line">. The results are displayed.

## Job list overview

The following job operations can also be performed from the VOD Job List:

* <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line"> **Start a new Job:** A new VOD job can be added.
* <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/fwiSgY3ETCRvugkHiAej/morelikethis.png" alt="" data-size="line"> **Start more like this:** A Job can be created, starting with duplicated settings from an existing Job.
* <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/LKZRuYLUpxkJ1ZQfgIJ4/disable.png" alt="" data-size="line"> **Disable Contents:** The contents of a VOD Job can be disabled which will mean they are not visible for playback in any of the V-Nova apps.
* <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/OAZZNBHeQpuQss2oSQMj/delete.png" alt="" data-size="line"> **Delete:** An existing job can be deleted.

## Start a new Job <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">&#x20;

1. Click the **Start a new Job** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">.
2. See [Create a VOD Job](/technologies/v-nova.platform/encoding/platform-api).

## Start more like this <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/fwiSgY3ETCRvugkHiAej/morelikethis.png" alt="" data-size="line">&#x20;

To start a new VOD Job, by duplicating the settings of an existing VOD job:

* In the **Job List**, check the checkbox to the left of the **Job ID** of the job that matches the specifications of the new job that is to be created.&#x20;
* Click the **Start more like this** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/fwiSgY3ETCRvugkHiAej/morelikethis.png" alt="" data-size="line">. A new encoding job is added to the jobs list called "Copy of \<duplicated job name>"
* Open the copied job and edit the settings as required as per [Creating a VOD (Video on Demand) Job](/technologies/v-nova.platform/encoding/platform-api).
* Click the **Encode**  icon. The new VOD Job encodes with the Encode status displayed in the **Status** column.

## View completed VOD Job details

To view the details of a VOD Job:

* In the **Job List**, click the **Job ID** of the VOD Job. The following is displayed:

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/gHWTarXYDmNfyIGWGWDh/VOD_job.png)

The following fields are displayed:

* **Title (mandatory):** The title of the VOD Job. This field can be edited.
* **Packaging:** Packaging, as set in the encode. Not editable.
* **Show on apps:** Enable - show on V-Nova applications with same login, i.e. web player.&#x20;
* **Content ID:** Assigned when the encode is performed. Not editable.

## Edit title

To edit the Title:

1. In the **Title** field, enter the new **Title**.
2. Click the **Save** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/oLZseBPKQsTnw0UCSKiE/save_icon.png" alt="" data-size="line">at the top right of the interface. This saves the updated settings.
3. Click the **BACK** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/XFn9jy4Zp4gSaw2kJVqH/return_icon.png" alt="" data-size="line"> to return to the **Job list**. The updated **Catalogue status** is displayed.

## Delete existing Job <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/OAZZNBHeQpuQss2oSQMj/delete.png" alt="" data-size="line">&#x20;

To delete a job:

* Select the job in the Job list, then click the **Delete existing Job** <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/OAZZNBHeQpuQss2oSQMj/delete.png" alt="" data-size="line"> icon. A job's **Catalogue status** must be **disabled** to be able to delete.
* When the confirmation popup appears: Click **Cancel** to cancel the operation, or **OK** to perform the operation. If **OK** has been selected, then the VOD Job is deleted from the Jobs List.

##


# Users and account details

The Users and My Account pages enable platform users to configure and manage their account.

## **Users**

**Administrator** users are able to access the list of all users in their account by selecting **System --> My Account** on the left-hand navigation panel. To add additional users to your account click the New User icon<img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">at the top right and complete the details. New users will receive an email containing a temporary password and link that they must click to confirm their email address and activate the account.

New users can either be created as **Administrators** for full access privileges or as a **User** with limited rights.&#x20;

## **My account**

To view the details of your own login, go to “**System --> My Account**” on the left-hand navigation bar to view your Username, User Group (whether you are an administrator, normal user or a trial user). You are also able to update your name information and reset your password as required.&#x20;

### **To change password:**

Select option from the drop-down of password change, the Only option is Permanent. This option will change the password permanently. Enter the new password meeting the requirements and Save.

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/9X24ORjROEXfwiZ1xRLo/user_info.PNG)

**To Change First name, Last name and Profile picture:**

**First name:** (Mandatory) Enter a valid first name of the user in the text-box.

**Last name:** (Mandatory) Enter a valid last name of the user in the text-box.

**Profile picture:** (Optional) The Profile picture can be a maximum of 240x240 in width and height or a maximum of 100 Kb. More than these values pictures can’t be uploaded. User can browse from local system to Upload.

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/oL1IgrGzTWs4iNtdqsHJ/user_info_update.PNG)

After making any changes click the Save icon and click “OK” to confirm or “Cancel” to leave the page without making changes.


# Encoding

The pages in this section guide you through creating VOD and live encoding jobs on the V-Nova platform and how to configure presets to accelerate your work.


# Create a VOD encoding Job

This page explains how to use the V-Nova platform to create VOD assets.

## Create a VOD Job

The following sections describe the controls that can be set, to customize a VOD encode to your requirements.

To create a **VOD Job**:

1. Go to either **Dashboard --> Total VOD assets**, or to **Encode Jobs --> VOD**. The **Job List** is displayed
2. Click the **Start a new Job** icon<img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">. The **VOD Job** page **Content information** panel is displayed. The **VOD Job content information** panel contains five or six tabs depending on the user level, all of which contain configuration parameters:
   * **General**
   * **Input**
   * **Profiles**
   * **Base codec**
   * **LCEVC**
   * **Advanced** (only when enabled by administrators)

The settings in each of the tabs are described in the following sections.

## **General tab**

![General tab](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/ssLQH81pH25ZfOXZ4V8U/general_tab.PNG)

'The **General** tab includes the following fields:

* **Title (mandatory):** Enter title
* **Template:** Load a pre-set template
* **Base encoder:** Define base encoder to use
* **Rate control:** Define rate control method to use. *Default is **CBR.***
* **Container:** Select one of the following:
  * **Fragmented MP4**
  * **TS chunks**
  * **M4S** (Live only)
  * **Webm** (only for VPX or AV1)
* **Segment length (mandatory):** Duration in seconds of packaged segments. *Default value is **10**.*
* **Audio:** Define Audio codec to be used. *Default is AAC for H264/H265, Vorbis for VPX + AV1.*
* **GOP:** Define GOP (Group-of-pictures) in seconds. *Default is **2.0** seconds.*
* **B-frames**: Define number of B-frames to be used (if relevant). *Default is **Auto**.*

## **Source tab**

One can browse and upload a video source file in a range of supported formats, or provide a public link to the source asset (e.g. from an AWS S3 bucket)

![Source tab](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/pew00OF2FRCiwbI99ztT/vod__input.png)

Enter the source information for either of the fields:

* **Source File:** Select a file to upload and encode from or,
* **Link to the Asset:** Enter a publicly accessible link to the desired asset.
* **Select example:** Ready to use demo sources.&#x20;

Input File requirements:

* File formats accepted: mp4, mov, mxf, avi, yuv\*.&#x20;
* Source file size must be under 15GB\*\**.*&#x20;

\* requires advanced encoding options to be enabled on your account. \
\*\* Contact us to enable encoding from larger files.

## **Profiles tab**

### **Overview**

The **Profiles** tab is the place to define the encoding **resolution** & **bitrate** or CRF target, or use the LCEVC Ladder builder provided. **Width, Height, Video bitrate** or CRF and **Audio Bitrate** are **mandatory** fields.&#x20;

Each **Profile** is part of the same adaptive streaming ladder, which is a range of different resolutions and bitrates that enable the player to dynamically adapt to the available internet bandwidth.

Upon selecting an existing preset, the encoding type and rate-control fields are automatically updated from the **General** tab.&#x20;

#### Profiles

A **Profile** includes the following settings:

* **Profile No:** The number of the **Profile** assigned to the **Preset's profile**. A **Profile number** is assigned in the order in which the **Profile** is created.&#x20;
* **Width:** Width of the video, in pixels, set for the **Profile** which can be edited as required.
* **Height:** Height of the video, in lines, set for the **Profile** which can be edited as required.
* **Video bitrate:** Video bitrate set for the **Profile** in kbps which can be edited as required.
* **Audio bitrate:** Audio Bitrate set for the **Profile** in kbps which can be edited as required.
* **Apply LCEVC:**
  * If enabled, then **LCEVC** is applied to the profile.
  * If disabled, then **LCEVC** is not applied to the profile and it will use the base codec natively.

A new **Profile** can be added as follows:

* In the **Add/Remove** column, click the **Add profile** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">. A new row is added, populated with the settings from the previous row, and assigned the next available **Profile number**. **Profile numbers** are not reused. If a **Profile** had been removed, then its number is no longer available. As such, the newly assigned number may skip one or more numbers.
* Edit the **Width**, **Height**, **Video bitrate**, **Audio bitrate** fields, and enable or disable Apply  **LCEVC**.

To remove a **Profile**:

* Click the **Remove profile** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/Mkewa5dE20wGx8nraN32/remove_-.png" alt="" data-size="line"> associated with the **Profile** that is to be removed.

![Basics tab](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/bY7xZKNVjfGNlcWxZDCW/vod_profiles.png)

The fields are described in the following sections.

### LCEVC ladder builder

#### Using LCEVC ladder builder

The LCEVC ladder builder is based on extensive video quality analysis for a range of use cases and content types and aims to guide you to profile settings that will deliver strong results. It is important to change resolution and bitrate settings when adopting LCEVC enhancement to achieve the best results. Please refer to [Best Practices](https://docs.v-nova.com/v-nova/lcevc/lcevc-best-practices) for further details.&#x20;

To use the builder, enter the details from each drop down menu that best represent your use case and press generate, the tool will automatically generate all of the profiles.&#x20;

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/CXaJ5qMWTcPVExtshTjA/lcevc_ladder_builder.PNG" alt=""><figcaption><p>LCEVC Ladder Builder</p></figcaption></figure>

### Templates

#### Selecting a template

Each **Template** provides typical recommended parameters to encode an asset. A template contains all of the settings associated with a job except for the source specific and its title. To select a built-in **Template**:

* From the **Template** dropdown, select a option. The **Profiles** associated with the selected **Templates** are displayed:

![Selecting a Template](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/fhIkwxs5nQmlnu8TtAyk/vod_presets.png)

Custom templates can also be saved at any time by selecting **Save as template** on the **General** tab for your job if you would like to recall those settings immediately for future jobs.

### **Base parameter tab**

The tab title and given parameters will change depending on the base encoder selected in the **General** tab. Settings will be left in default if not specified by the **Template** or **Ladder Builder**.&#x20;

![x264 tab](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/npsU5866bjOYYfBCWzx7/base_params.png)

This area in the UI enables adjustement of the common parameters for the base encoder that LCEVC is enhancing. If you require more please contact <platform@v-nova.com> to enable the advanced parameter tab.&#x20;

### **LCEVC parameter tab**

This tab is to control the LCEVC tools. Tools are best left in default 'Auto'.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/DhHW20TV8ZMUoykmZBaj/lcevc_params.PNG" alt=""><figcaption></figcaption></figure>

**LCEVC settings**

* [**LCEVC tune**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#lcevc-tuning)
  * Video Quality
  * Animation
  * VMAF
  * SSIM
  * PSNR
* [**LCEVC preset**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#lcevc-preset)
  * 0-5
* [**Scaling mode**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#v-nova-lcevc-scaling-mode)
  * Auto
  * 2D: Force 2D scaling thoughout profiles
  * 1D: Force 1D scaling throughout profiles
* **Transform Type**
  * **Auto:** This setting is determined intelligently by the encoder based on the LCEVC **Tune** setting. &#x20;
  * Double Directional
  * Directional
* [**M Adaptive Downsampling**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#m-adaptive-downsampling)
  * **Modes:**
    * **Auto:** Automatic **M Adaptive Downsampling** is applied.
    * **Replace:**&#x20;
    * **Seperate:**
    * **Off:** Force off.&#x20;
  * &#x20;[**Strength**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#m_hf_strength)**:** Set via spinner or direct entry.
    * If **Mode** is set to **Replace or Separate**, then is grayed out.
    * If **Mode** is set to **Replace or Separate,** then **Range: 0 - 0.5**.
* [**Dither**](https://docs.v-nova.com/v-nova/lcevc/reference-applications/ffmpeg#dithering)
  * **Apply dither**
    * **Auto:** Automatic **Dithering** is applied and is dictated by the LCEVC tuning employed.
    * **Force On:** Dithering is determined by user-set **Strength** setting.
  * **Strength**
    * If **Apply Dither** is set to **Auto**, then Is grayed out.
    * If **Apply Dither** is set to **Force On**, then **Range: 1 - 10**.

### **Extra tab**

This tab is avaliable to accounts or users upon request. Parameters should be entered as if raw into a FFmpeg command line. LCEVC's EIL parameters must have **';'** after each one, including the final.&#x20;

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/LpEDBe471WpiTRRohjUH/Extra_tab.PNG" alt=""><figcaption></figcaption></figure>

## Performing the encode

To perform the encode and view the resulting content, proceed to [Performing the encode and viewing content](/technologies/v-nova.platform/encoding/performing-the-encode-and-viewing-content).


# Creating encoding templates

This page describes functionalities of presets that can be used in encoding VOD jobs and Live Channels.

## **VOD and Live Templates**

Whilst the encoding parameters are largely the same for VOD and Live encoding, a template created can be used for either job type.&#x20;

### **Template Creation**

Template are made up of one or more profiles associated with a specific encoding type and the individual encoding parameters for each. The V-Nova platform also includes a selection of default presets which can be used to get started. For advice on optimising ABR ladders when applying LCEVC enhancement to your workflows please [contact us](https://docs.v-nova.com/technologies/v-nova.platform/encoding/platform-api/www.v-nova.com/contact).

To add a **Template**:

* From the left navigation, go to **Configuration Presets --> Template**, and then click the **Add a new Template** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/ECmwThswDnaNmPhPw0iZ/templates_1.PNG" alt=""><figcaption></figcaption></figure>

The **Template** page include the following tabs, all of which contain configuration parameters:

* **General**
* **Profiles**
* **Base encoder**
* **LCEVC**
* **Extra** (If enabled)

The settings in each of the tabs are described in the following sections.

## **General tab**

![VOD Preset General tab](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/4UOLSDjxntVkQncDd48I/template_general.PNG)

**Title (Mandatory):** Enter a title for the **Template**.

* **Template Access (Account level only):**
  * **Individual:** Template is only available to the user creating the template
  * **For Account:** Template is available to all users in the account

**Note for Trial Users:** Presets created by a Trial User can only be used  by the Trial User who created it.

## **Setting up your Template**

Once you have specified the title and access settings for your template, the remaining settings are exactly the same as creating a job manually.&#x20;

See [Create a VOD Encoding Job](/technologies/v-nova.platform/encoding/platform-api) and [Set up a Live Channel](/technologies/v-nova.platform/encoding/platform-live-channel-creation) for more details.


# Set up a Live channel

This page describes live channel creation and its features. Please note that live channel encoding is not available for trial users.

## **Live channels interface**

To Encode a live channel in the platform:

* Go to **Encode Jobs --> Live**. The **Live channels channel list** is displayed.

![Live Channels Channel List](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/qm1EgEvqH1fXx23KCSbl/live_dashboard.PNG)

&#x20;This page lists all of the encoded Live Channels and displays the details of the following:

* **Stream ID:** The **Stream ID** can be sorted in ascending or descending order.&#x20;
* **Channel name:** Name of the Live Channel.
* **Content ID:** ID assigned by the platform when the Live Channel is created.
* **Copy manifest URL:** Manifest URL that is associated with the Live Channel.
* **Profiles:** Number of profiles associated with the Live Channel.
* **User:** Specifies the user name of the account that created or last edited the channel.
* **Status:** Specifies if the Live Channel is enabled or disabled.
* **Edit:** Opens the channel to enable editing of its parameters or to view realtime information about the status of an enabled active channel.

A user can:

* Encode a Live Channel and live stream.
* Edit an existing Live Channel.
* Delete an existing Live Channel.

## **Creating a new Live channel**

To create a new **Live channel**:

&#x20;Click the **Start a new Live stream** icon <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/HL5GfjO4DBLzNus82Cp6/new_+.png" alt="" data-size="line">. The **Live stream configuration** page is displayed.

The **Live Job** page includes the following tabs, all of which contain configuration parameters:

* **General**
* **Input**
* **Profile**
* **Base codec**
* **LCEVC**
* **Extra**

The settings in each of the tabs are described in the following sections.

### General tab

The **General** tab specifies Live Channel set-up information. Most fields require input, and are indicated as mandatory by red asterisks.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/8IbnTgqb55ynl2TvkBIH/LIVE_general_2.PNG" alt=""><figcaption></figcaption></figure>

* **Job name (mandatory):** Enter a title for the Live Channel.
* **Template:** Select an encoder job configuration template&#x20;
* **Base encoder:** Select base encoder to (optionally) be enhanced by LCEVC
* **Rate control:** Select the rate-control mode for the job (either CBR of pCRF)
* **Package as:** TS Chunks (Transport Stream) or M4S Chunks (Mp4 chunks)
* **Segment length (mandatory):** Duration in seconds of packaged segments. *Default value is **10**.*
* **Audio:** Define Audio codec to be used. *Default is AAC for H264/H265, Vorbis for VPX + AV1.*
* **GOP:** Define GOP (Group-of-pictures) length in seconds. *Default is **2.0** seconds.*
* **B-frames**: Define number of B-frames to be used (if relevant). *Default is **Auto**.*

### Input

This tab allows you to specify the input URL of your live source content in RTMP format. This can be pulled from the V-Nova RTMP server or pushed to with a custom generated URL.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/nNmkTLFYNA8Fv2YNC3ne/live_source_2.PNG" alt=""><figcaption></figcaption></figure>

* **Type of RTMP (Real-time Messaging Protocol) Source (mandatory):** Select the RTMP type from the drop-down.
  * **V-NOVA RTMP SERVER:** If the V-NOVA RTMP SERVER is selected, then the LCEVC platform automatically generates the RTMP URL.
  * **CUSTOM RTMP SERVER:** If the CUSTOM RTMP SERVER is selected, then the URL must be specified.
* **Choose Zone for RTMP endpoint (mandatory for V-Nova RTMP server):** Select the Zone for the RTMP endpoint from the drop-down.
  * **Europe - London**
  * **US - East-North Virginia**
* **RTMP URL (mandatory for custom RTMP server):** Enter a valid URL in the text-box.

### Profile

The profile tab is the same as used in [VOD jobs](https://docs.v-nova.com/v-nova/v-nova-platform/encoding/platform-api), a user must define all fields and profiles, or use the LCEVC Ladder builder. The platform will automatically create a cloud server instance with the required processing capacity to encode the profiles and codec parameters chosen.&#x20;

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/EBEk0NkSBSSuvhm1WvIN/live_profiles_2.PNG" alt=""><figcaption></figcaption></figure>

#### Base parameter tab

Similar to the VOD encoding parameters for x264/x265, the live x264/x265 encoding has the same set of parameters. The tab title and given parameters will change depending on the base encoder selected in the **General** tab. Settings will be left in default if not specified by the **Template** or **Ladder Builder**.&#x20;

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/RrmuibMC8MuQ6NlxiRmT/live_base_2.PNG" alt=""><figcaption></figcaption></figure>

This tab enables adjustment of the common parameters for the base encoder that LCEVC is enhancing. If you require more, please make a request to enable the advanced parameter tab with a support query through the support link on the platform.

#### LCEVC parameter tab

This tab controls the LCEVC tools. The default value for most of them is "Auto" which allows the LCEVC encoder to intelligently determine the best value depending on the other parameters of your encode job.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/4IzXSwUXNOaPCd7ZCdth/live_lcevc_2.PNG" alt=""><figcaption></figcaption></figure>

#### Extra tab for additional parameters

*Only avalaible to those with advanced encoding parameteres enabled.* \
Parameters should be entered as if raw into a FFmpeg command line. LCEVC's EIL parameters must have **';'** after each one, including the final command specified.&#x20;

![Live Job - Extra tab for H264/265](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/3ELHAocc57PEpmE4nzLB/live_extra.PNG)

#### Start live encoding

Once you have set all your parameters, you can either save as a draft or start the job by pressing 'Encode Live'. Once this is pressed it will go into a queue to launch an instance upon which to encode the channel.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/U5m1BrW2evcaa4MgJKLN/live_save.PNG" alt=""><figcaption><p>Save Draft or Encode Live Button</p></figcaption></figure>

Once either button is pressed you will return to the Live dashboard where you can monitor the status of the job.&#x20;

## **Modifying an existing Live channel**

To modify an existing Live channel:

Click the **Edit** icon, you will see the details of options selected before. Additionally a User is presented with a statistics page to see the health of the instance.

![Live Statistics Page](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/5QRTt6YMeKVp7cn6ViOT/live_statistics_2.PNG)

In order to modifying a Live channel you must first disable the job from the General tab. If the channel is active, the majority of controls will be disabled.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/SrvCz5O1q2GbiZBMHyj8/live_settings_running.PNG" alt=""><figcaption><p>Locked fields as job is running</p></figcaption></figure>

Any settings or fields can be updated and saved. Once all changes have been made you can Enable the Transcoding once again from the General tab.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/gIaXLC4EW8i3I7um6vBJ/live_reenable.PNG" alt=""><figcaption><p>General tab to re-enable a stream</p></figcaption></figure>

### **Deleting an existing Live channel**

To delete an existing Live channel:

Click on the check-boxes for any existing Live channels that need to be deleted, and click on the “Delete” icon. A confirmation dialogue box will be shown. Click on “OK” to delete or click on “cancel” to avoid deletion. Channels must be disabled before they can be deleted.

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/oZzhsF0Hs8rRy0BIKgxl/live_delete.PNG)


# Perform the encode and view content

Performing the encode

After all values have been set to perform the encode:

* Click the **Encode** <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/FVzXPO5NWdXHE4tkqinH/encode.PNG" alt="" data-size="line"> button.

### Uploading a source

If uploading a file, the page will return to the input tab and upload status will be shown. Once complete, the page will revert to VOD dashboard as the job enters the queue for encoding.&#x20;

![Encode progress](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/FBNBXvYr46r4XLbXvKFh/source_upload.PNG)

### Encode status

A user can wait and revert, or monitor the status from the **General** tab within the **Job ID**.

<figure><img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/FWfFyJKOSGlYK5jGZuHC/encode_progress.PNG" alt=""><figcaption></figcaption></figure>

Once encoding is complete, then the **Job ID**, **Job Names**, **Content ID**, **Manifest URL**, **Duration**, , **Catalogue Status**, **Status** and **Source** are displayed on the **Job List**, accessed via **Encode Jobs --> VOD** page.

![Completed job details](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/mp6xJCOLvOfH5ufSSz60/vod_dashboard.png)

Encode Status specifies whether or not the encode was successful and gives other information during the encoding process to indicate what stage the encode is at.

* If the encode was successful, then the **Status** displays **Encode Successful**.
* If the encode was not successful, then the **Status** displays **Error Warning**.

The encode log displays information about the encode. To display the encode log:

* Go to **Encode Jobs --> VOD --> Logs tab**.

![Log details](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/dlwYVQUaW5rMpOYfnAm9/vod_logs.PNG)

### Delete content

To delete a content item:

1. Go to **Encode Jobs --> VOD**.
2. The Job must be in a disabled state, this can be done by selecting the job and pressing  <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/LKZRuYLUpxkJ1ZQfgIJ4/disable.png" alt="" data-size="line">&#x20;
3. Check the checkbox to the left of the **Job ID** for the content item that is to be deleted.
4. Click the **Trashcan** <img src="https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/OAZZNBHeQpuQss2oSQMj/delete.png" alt="" data-size="line"> icon.

![Job list](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/dZTQKITJOhwpaffYb2OB/vod_disabled.PNG)

## **Viewing my content**

Once an encode has successfully been completed, the content can be viewed by logging into the V-Nova application available from the relevant app stores (Android/iOS/Windows) or on web by visiting <https://player.v-nova.com>.

If you wish to enable LCEVC-enhanced playback in any of your own applications contact V-Nova to discuss.


# Platform API

This page provides a reference to the V-Nova Platform for encoding LCEVC-enhanced x264 and x265 encodes using the REST API .

## Before you start

The Platform’s REST API is designed to rapidly create automated encoding solutions and to facilitate the integration of LCEVC-enhanced encoding into other services and platforms. To learn more about the options for integrating LCEVC contact V-Nova at [www.v-nova.com/contact](https://docs.v-nova.com/technologies/v-nova.platform/encoding/www.v-nova.com/contact).

This section details how to use the platform’s API to retrieve encode information from a previous job or start a new job with defined parameters for VOD or live content.

In order to authenticate your user account with the platform's API you must acquire an API key. Please bear in mind this key will give full access to your account and it is therefore recommended to store the key in a secure location to guard against the potential security risk of exposing an account API key to third parties. The platform API only supports JSON.

### To access your API key

1. Login into the platform, then go to **System** on the left-hand navigation panel and select **Users**.
2. Edit the user you wish to assign API access to. At the bottom of the user page you will find the API Key for that user.&#x20;
3. If the API key has not been generated, then go to **API Key Refresh,** select **Generate and save the user.** You may need to logout and back in again to retrieve the key.

## Get VOD asset details

<mark style="color:blue;">`GET`</mark> `https://api.v-nova.com/production/prod_vod/v-nova/{username}/{contentid}`

This will allow you to get the details of your account's encodes, if you specify a content ID it will only pull back the information of that particular job, if not it will pull back information on all of the encodes. \
Usign content ID is a GET method only.

#### Headers

| Name   | Type   | Description                         |
| ------ | ------ | ----------------------------------- |
| apikey | string | Key generated by platform for user. |

{% tabs %}
{% tab title="200 VOD Listing successful" %}

```
{
    "Message": "VOD Listing Successful ",
    "Username": "Demo",
    "Total VOD Contents": 1,
    "EncodingParameters": {
        "packaging": {
            "Description": "Packaging Options available on the platform",
            "Values": {
                "Hls_ts": "0",
                "Dash_fmp4": "1"
            }
        },
        "encodingType": {
            "Description": "Encoding Engine to be used for the encoding of this profile",
            "Values": [
                "2",
                "3"
            ]
        },
        "width": {
            "Description": "width of the encoded video for this profile"
        },
        "height": {
            "Description": "height of the encoded video for this profile"
        },
        "transform": {
            "Description": "specify the transform type for encoding.",
            "Values": [
                "dd",
                "dds"
            ]
        },
        "ditherStrength": {
            "Description": "setting strength of dither"
        },
        "x264Preset": {
            "Description": "setting preset parameters for x264",
            "Values": [
                "very fast",
                "fast",
                "medium",
                "slow",
                "very slow",
                "placebo",
                "dual pass"
            ]
        },
        "QpMin": {
            "Description": "Advanced perseus parameter forspecifying the minimum quanatization for the encoding in thisprofile."
        },
        "QpMax": {
            "Description": "Advanced perseus parameter forspecifying the maximum quanatization for the encoding in thisprofile."
        },
        "maxgop": {
            "Description": "maximum no of gop present in a chunk"
        },
        "videoBitrate": {
            "Description": "video bitrate per second for the video profile"
        },
        "audioBitrate": {
            "Description": "audio bitrate per second for the video profile"
        }
    },
    "VodAssets": {
        "catalogue:": [
            {
                "Title": "XXX",
                "ContentId": "XXXXXX",
                "ContentDuration": "00:00:10.00",
                "IsPremium": false,
                "IsEnabled": true,
                "AdURL": "https://s3.eu-west-2.amazonaws.com/XXXX",
                "Packaging": "HLS - ts chunks",
                "ManifestURL": "https://s3.eu-west-2.amazonaws.com/origin-lon.v-nova.com/XXXX",
                "ThumbnailImage": "https://s3.eu-west-2.amazonaws.com/origin-lon.v-nova.com/images/XXXX",
                "PosterImage": "https://s3.eu-west-2.amazonaws.com/origin-lon.v-nova.com/images/XXXX",
                "Description": "",
                "Cast": "",
                "NoofProfiles": "1",
                "Chunksize": "10",
                "Profiles": [
                    {
                        "Resolution": "1920X1080",
                        "Bitrate": "1024000",
                        "Maxgop": "50",
                        "ProfileNo": 1,
                        "Size": "1.250000",
                        "Audiobitrate": "64"
                    }
                ]
            }
        ]
    }
}
```

{% endtab %}

{% tab title="411 Database error, incorrect contnet ID" %}

```
{
    "Error Code": "411 ",
    "Message": "Database Error ",
    "Username": "Demo ",
    "Query Error": "sql: no rows in result set ",
    "Error": "There was an issue fetching the data from the database. 'XXXXX' does not exist."
}

```

{% endtab %}
{% endtabs %}

## Get Preset Parameters

<mark style="color:blue;">`GET`</mark> `https://api.v-nova.com/production/prod_vod/presets/{presetID}`

`/presets` returns a list of presets and their parameters. \
\
`/presets/{presetID}` retrieves the preset's parameters that can be used to populate all parameters in a post job.&#x20;

#### Path Parameters

| Name   | Type   | Description                     |
| ------ | ------ | ------------------------------- |
| apikey | string | Key generated by platform user. |

{% tabs %}
{% tab title="200 " %}

```
{
    "Message": "VOD Presets Listing Successful ",
    "Total Presets ": 1,
    "VodPresets": {
        "catalogue:": [
            {
                "PresetID": "200",
                "Title": "720p_50/60_Single_HQ",
                "NoofProfiles": "1",
                "Profiles": [
                    {
                        "EncodingType": "2",
                        "ApplyRateControl": "0",
                        "Width": "1280",
                        "Height": "720",
                        "VideoBitrate": "1740",
                        "AudioBitrate": "96",
                        "ApplyLcevc": "on",
                        "X264Preset": "medium",
                        "X264RateControl": "cbr",
                        "X265Preset": "",
                        "X265RateControl": "",
                        "LcevcTune": "vq",
                        "ScalingMode": "auto",
                        "Transform": "auto",
                        "Bframes": "3",
                        "Maxgop": "50",
                        "MAdaptiveDownsampling": "auto",
                        "HFstrength": "NA",
                        "LFstrength": "NA",
                        "DitherCheck": "auto",
                        "Strength": "NA",
                        "QpMin": "NA",
                        "QpMax": "NA",
                        "AdditionalParameters": "-pix_fmt yuv420p ",
                        "AdditionalEILParameters": "accel_num_worker_threads=8;rc_pcrf_base_rc_mode=cbr;lcevc_tune=vq;preset=medium;"
                    }
                ]
            }
        ]
    }
}



```

{% endtab %}
{% endtabs %}

## Post Offline file (VOD) - Using parameters&#x20;

<mark style="color:green;">`POST`</mark> `https://api.v-nova.com/production/prod_vod/v-nova/{username}`

The body of a post must take the form below. If a presetID is set the profile parameters do not need to be defined. See next chapter. We have set default values in the below body. If you wish to generate more than one profile, simply duplicate the profile parameters in the same POST string.

#### Headers

| Name   | Type   | Description               |
| ------ | ------ | ------------------------- |
| apikey | string | Key generated by platform |

#### Request Body

| Name                | Type    | Description                                                                                                                              |
| ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| title               | string  | Title for encode job                                                                                                                     |
| url                 | string  | URL of source.                                                                                                                           |
| cast                | string  | Text description.                                                                                                                        |
| description         | string  | Text description.                                                                                                                        |
| Rating              | string  | Audience rating, text field.                                                                                                             |
| segmentDurationSecs | number  | GOP Segment duration. (Default is '10')                                                                                                  |
| encodingMode        | boolean | Define which codec to use. \[Values '0' MPEG-5 LCEVC x.264, '1' MPEG-5 LCEVC x.265] (Default '0')                                        |
| packagingMode       | boolean | \[Values '0' HLS with transport stream base, '1' DASH with fragmented mp4]. (Default '1')                                                |
| presetID            | number  | Numerical ID of the preset as defined per user.                                                                                          |
| encodingType        | boolean | AWS encoding engine to be used. \['2' ECS Fargate, '3' EC2] (Default EC2)                                                                |
| applyRateControl    | boolean | \[Values 0/1]                                                                                                                            |
| width               | number  | Pixel count of the width of the output video. (Default is 1920)                                                                          |
| height              | number  | Pixel count of the height of the output video. (Default is 1080)                                                                         |
| videoBitrate        | number  | Video bitrate defind in Kbps. (Default is 1000)                                                                                          |
| audioBitrate        | number  | Audio bitrate defined in Kbps. (Default 64Kbps)                                                                                          |
| applyLcevc          | boolean | To set a LCEVC or native encode. \[Values on/off] (Default on)                                                                           |
| x26\*Preset         | boolean | Preset parameters for x264 or x265 as defined. \[Values ery Slow, Slow, Very fast, Fast, Medium, Placebo, Dual-Pass] (Default is Medium) |
| x26\*RateControl    | boolean | Rate control for either x264 or x265 as defined. \[Values cbr & crf] (Default is crf)                                                    |
| lcevcTune           | boolean | Values vq, vmaf, ssim, psnr. (Default is ‘vq’)                                                                                           |
| transform           | boolean | Specify the transform type for encoding. Values \[dd and dds]. Default value is dds.                                                     |
| Scaling mode        | boolean | auto, 1D 2D (Default 2D)                                                                                                                 |
| Bframes             | number  | <p>How many B-Frames in the IPB GOP based structure. <br>\[Values 1-10] (Default is 3)</p>                                               |
| maxgop              | number  | Maximum value GOP present in a chunk. (Default is 50)                                                                                    |
| ditherCheck         | boolean | (Value on or off)                                                                                                                        |
| Strength            | number  | Wetting strength of dither. \[Value 1-10] (Default is 5)                                                                                 |
| QpMin               | number  | Advanced parameter for specifying the maximum quantization for the encoding in this profile. \[Value 1-51] (Default 25)                  |
| QpMax               | number  | Advanced parameter for specifying the maximum quantization for the encoding in this profile. \[Value 1-51] (Default 30)                  |

{% tabs %}
{% tab title="200 " %}

```
***************************************************************************
Response:-
***************************************************************************
{
        "title" : " API Document test  ",
        "url" : "http*****.mp4",
        "cast" : "",
        "description" : "",
        "rating" : "",
        "packaging" : "1",
        "segmentDurationSecs" : "10",
        "encodingMode" : "0",
        "packagingMode" : "0",
        "presetID" : "",
        "profiles" : [
            {
                "encodingType" : "2",
                "applyRateControl" : "0",
                "width" : "1920",
                "height" : "1080",
                "videoBitrate": "1000",
                "audioBitrate" : "64",
                "applyLcevc" : "on",
                "x264Preset" : "veryslow",
                "x264RateControl" : "crf",
                "x265Preset" : "",
                "x265RateControl" : "",
                "lcevcTune"   : "vq",
                "transform" : "auto",
                "scalingMode" : "auto",
                "bframes"     : "3" ,
                "maxgop" : "50",
                "mfilters" : "auto",
                "filter1strength" : "",
                "filter2strength" : "",
                "MAdaptiveDownsampling" : "",
                  "HFstrength" : "",
                  "LFstrength" : "",
                "ditherCheck" : "auto",
                "strength" : "5",
                "QpMin" : "25",
                "QpMax" : "30"

            }],

```

{% endtab %}

{% tab title="410 Authorisation failure" %}

```
{
    "Error Code": "410 ",
    "Message": "Authorisation failure ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="411 Database Error" %}

```
{
    "Error Code": "411 ",
    "Message": "Database Error ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="412 Parsing Error" %}

```
{
    "Error Code": "412",
    "Message": "Parsing Error ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="413 Missing fields or value in a field " %}

```
{
    "Error Code": "413",
    "Message": "Message ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="414 Invalid entry" %}

```
{
    "Error Code": "414",
    "Message": "Invalid Entry ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="415 Error in value of encoding type" %}

```
{
    "Error Code": "415",
    "Message": "Message ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}

{% tab title="417 Invalid input" %}

```
{
    "Error Code": "417",
    "Message": "Message ",
    "Username": "Demo ",
    "Error": "Error description"
}
```

{% endtab %}
{% endtabs %}

## Post Offline file (VOD) - Using Preset ID

<mark style="color:green;">`POST`</mark> `https://api.v-nova.com/production/prod_vod/v-nova/{username}`

Post offline file (VOD) with Preset's.

#### Path Parameters

| Name   | Type   | Description |
| ------ | ------ | ----------- |
| apikey | string |             |

{% tabs %}
{% tab title="200 " %}

```
***************************************************************************
Response:-
***************************************************************************
{
        "title" : " API Document test  ",
        "url" : "http*****.mp4",
        "cast" : "",
        "description" : "",
        "rating" : "",
        "packaging" : "1",
        "contentCategoryID" : "77",
        "segmentDurationSecs" : "10",
        "encodingMode" : "0",
        "packagingMode" : "0",
        "presetID" : "",
```

{% endtab %}
{% endtabs %}

## Example Post Command

An example of creating a ladder of two encodes from the same source file.&#x20;

```
{
    "title" : "ABC1",
    "url" : "input.mp4",   
    "cast" : "AC",
    "description" : "DE",
    "rating" : "AA",
    "packaging" : "1"
    "segmentDurationSecs" : "10",
    "encodingMode" : "0",
    "packagingMode" : "0",
    "presetID" : "",
    "profiles" : [
        {
            "encodingType" : "2",
            "applyRateControl" : "0",
            "width" : "1920",
            "height" : "1080",
            "videoBitrate": "1000",
            "audioBitrate" : "64",
            "applyLcevc" : "on",
            "x264Preset" : "veryslow",
            "x264RateControl" : "crf",
            "x265Preset" : "",
            "x265RateControl" : "",
            "lcevcTune"   : "vq",
            "transform" : "auto",
            "scalingMode" : "auto",
            "bframes"     : "3" ,
            "maxgop" : "50",
            "MAdaptiveDownsampling" : "auto",
            "HFstrength" : "",
            "LFstrength" : "",
            "ditherCheck" : "auto",
            "strength" : "",
            "QpMin" : "",
            "QpMax" : ""
        },
        {
            "encodingType" : "2",
            "applyRateControl" : "0",
            "width" : "1920",
            "height" : "1080",
            "videoBitrate": "1000",
            "audioBitrate" : "64",
            "applyLcevc" : "on",
            "x264Preset" : "veryslow",
            "x264RateControl" : "crf",
            "x265Preset" : "",
            "x265RateControl" : "",
            "lcevcTune"   : "vq",
            "transform" : "auto",
            "scalingMode" : "auto",
            "bframes"     : "3" ,
            "maxgop" : "50",
            "MAdaptiveDownsampling" : "auto",
            "HFstrength" : "",
            "LFstrength" : "",
            "ditherCheck" : "auto",
            "strength" : "",
            "QpMin" : "",
            "QpMax" : ""
        }
        ],

```

## Support

Errors are returned within standard http syntax, these are shown above in the response tab of the GET or POST headers. If you experience any errors or wish to file a bug report, please contact your V-Nova representative or <support@v-nova.com> and include the API response concerned.&#x20;


# Decoding

Playing back content that you have encoded on the V-Nova platform is very straightforward as the platform is fully integrated with the V-Nova demo applications which are available from all of the major app stores. Simply sign into the V-Nova demo application using your V-Nova Platform credentials and all of your content will be listed for playback directly within the app.

Search "V-Nova" on the relevant store or follow the links below:

![](https://content.gitbook.com/content/xdWpFEYGHZ7YDDpkZo7J/blobs/28T0r1QGufbU3U7N2MRf/image.png)

[Download here from the Apple AppStore](https://apps.apple.com/gb/app/v-nova/id1363898377)

[Download here from the Google PlayStore](https://play.google.com/store/apps/details?id=com.vnova.PERSEUS\&hl=en_GB\&gl=US)

[Download here from the Microsoft Store](https://www.microsoft.com/en-qa/p/v-nova/9n9c2z21xhl9?cid=msft_web_chart\&activetab=pivot:overviewtab)

[A website player is also available here.](https://player.v-nova.com)

### Integrating playback in your applications

LCEVC-enhanced decoding is deployable in software at a variety of levels in your software stack. After testing encoding and playback from the V-Nova Platform and associated demo apps, should you wish to proceed with testing integration within your own players please review the documentation sections on our [Reference Integrations](/technologies/mpeg-5.lcevc/integrations/integrations) and [Decoder Integration Layer](/technologies/mpeg-5.lcevc/decoder/sdk/decoder-integration-layer-dil) and [contact us](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/) for more details.

A range of other decoding implementations are also available including AOSP and MFT. [Contact us](https://www.v-nova.com/london-based-ip-and-software-company/contact-v-nova/) to find out more.


# Welcome

ImmersiX is a multi-content app poised to revolutionize visual storytelling across any OpenXR headset, including the Meta Quest. ImmersiX seamlessly blends the creative essence of immersion with the scientific power of six degrees of freedom (6DoF), bridging the worlds of art, science, and technology in ways never seen before.

This next-generation app not only showcases original movies from V-Nova Studios, but also opens the door for third-party creators to add their own immersive experiences, making ImmersiX a dynamic hub for world-class XR storytelling. Audiences are transported beyond imagination, exploring breath-taking worlds with complete freedom and presence.

Powered by the ground-breaking V-Nova PresenZ format, ImmersiX delivers hyper-realistic visuals, offering an unprecedented, authentic, sense of movement and detail that rivals reality itself. Coupled with spatial audio, the experience becomes truly lifelike, pushing the boundaries of what is possible in entertainment and technology.

ImmersiX is more than an experience — it is a new frontier in storytelling, where viewers don’t just watch — they feel, explore, and connect like never before.

Find out more about ImmersiX and V-Nova Studios:

{% embed url="<https://v-nova.com/studios>" %}


# Getting started

{% hint style="warning" %}
Before starting, please ensure your device meets the [System requirements](/technologies/immersix/system-requirements).
{% endhint %}

## Step-by-step guide

This guide provides step-by-step instructions for setting up a Meta Quest headset and launching the ImmersiX application. For further information and troubleshooting your connection to the Meta Quest headset, please refer to the Meta Quest [Help Centre](https://www.meta.com/en-gb/help/quest/).

This guide focuses on the setting up a Meta Quest headset as an example. However, the ImmersiX application can also work with other OpenXR compatible headsets.

{% stepper %}
{% step %}

### Download Steam

Download and install Steam from [here](https://store.steampowered.com/about/). Create a new Steam account or sign in with an existing one.
{% endstep %}

{% step %}

### Download ImmersiX

You can download the ImmersiX application from the Steam Store [here](https://store.steampowered.com/app/1704450/ImmersiX/).
{% endstep %}

{% step %}

### Download Meta Quest Link

Download and install Meta Quest Link by following the instructions [here](https://www.meta.com/en-gb/help/quest/articles/headsets-and-accessories/oculus-rift-s/install-app-for-link/). Create a new Meta account or sign in with an existing one.
{% endstep %}

{% step %}

### Set-up Meta Quest headset

Open the Meta Quest Link application. If this is the first time you are using your Quest headset, please follow the onscreen instructions to set-up your new headset. You may need to download the Meta Horizon mobile application to complete the set-up of your Quest headset. Further instructions to set-up your headset can be found in the Meta Quest Help Centre ([Quest 3S](https://www.meta.com/en-gb/help/quest/articles/getting-started/getting-started-with-quest-3s/set-up-your-meta-quest-3s/), [Quest 3](https://www.meta.com/en-gb/help/quest/articles/getting-started/getting-started-with-quest-3/set-up-quest-3/), [Quest Pro](https://www.meta.com/en-gb/help/quest/articles/getting-started/getting-started-with-quest-pro/pro-basics/), [Quest 2](https://www.meta.com/en-gb/help/quest/articles/getting-started/getting-started-with-quest-2/set-up-meta-quest-2/)).&#x20;

{% hint style="info" %}
If you are experiencing connection issues, we commend using *Link* with a USB-C cable (instead of *AirLink*) to connect your Quest headset to your PC. More information can be found in the Meta Quest [Help Centre](https://www.meta.com/en-gb/help/quest/).
{% endhint %}
{% endstep %}

{% step %}

### Set up Meta Quest Link (application)

Go to Settings -> General in the Meta Quest Link application on your PC.&#x20;

1. Allow "Unknown Sources"
2. Set Meta Quest Link as active OpenXR runtime

<img src="https://1563923258-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F14O9SSDxghEHEuA3ThIX%2Fuploads%2FfQ1pXsNlgNaVcmXRElew%2FMeta%20Quest%20Link%20settings%20crop%20(Large).png?alt=media&amp;token=3cebb8e7-eb8d-4197-852e-e7465e46942d" alt="" data-size="original">
{% endstep %}

{% step %}

### Launch Meta Quest Link (headset)

In your headset, launch the Quest Link. Further information can be found [here](https://www.meta.com/en-gb/help/quest/articles/headsets-and-accessories/oculus-link/connect-with-air-link/).

1. Press ![Meta button for VR headset controllers](https://static.xx.fbcdn.net/assets/?revision=1087642415887863\&name=oculus-metabt-icon\&density=1) / ![oculus icon](https://static.xx.fbcdn.net/assets/?revision=1087642415887863\&name=oculus-oculus-icon\&density=1) on your controller to open the universal menu.
2. Select Settings, then System, then select Quest Link.
3. Select the computer that your headset is connected to, then select Launch.
   {% endstep %}

{% step %}

### Launch ImmersiX :tada:

Launch the ImmersiX application from your Steam library.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
Question? We're happy to help! Please [Contact us](/technologies/immersix/contact-us).
{% endhint %}


# System requirements

When using the ImmersiX application, your desktop or laptop should meet the below minimum system requirements.

* OS: Windows 10 64-bit (Home or Pro) or newer
* Processor: Intel Core i7-12700KF / AMD Ryzen 7 5800X3D
* Memory: 16 GB RAM
* Graphics:&#x20;
  * NVidia RTX 2080Ti or better
  * AMD Radeon RX 7700 XT
* DirectX: Version 12
* Storage: 30 GB available space&#x20;
* VR Support: Any OpenXR headset (Meta Quest 2, 3, or 3S recommended)\ <br>


# Frequently asked questions

Our Frequently Asked Questions (FAQs) section addresses the most common inquiries about ImmersiX. Here, you'll find answers to questions about upcoming immersive movie releases, application storage requirements, and more. If you have further questions or need additional information, feel free to [Contact us](/technologies/immersix/contact-us).

<details>

<summary>What is ImmersiX?</summary>

ImmersiX is a multi-content app poised to revolutionize visual storytelling across any Open XR headset, including the Meta Quest. ImmersiX seamlessly blends the creative essence of immersion with the scientific power of six degrees of freedom (6DoF), bridging the worlds of art, science, and technology in ways never seen before.

</details>

<details>

<summary>Will more movies be released on ImmersiX?</summary>

More content will be released on ImmersiX in the near future. Keep an eye on our [Steam page](https://store.steampowered.com/app/1704450/ImmersiX/) or follow us on our social media channels to be among the first to try new content.&#x20;

</details>

<details>

<summary>Do I need a VR headset to use ImmersiX?</summary>

Yes, an OpenXR compatible headset is required for playing immersive movies through ImmersiX.

</details>

<details>

<summary>Why does the application require much SSD storage space?</summary>

Immersive content requires more storage than regular 2D or even 360° movies as the entire scene is captured and stored as a point cloud. We are continuously improving our technology and already use various optimisations to keep the required storage as small as possible.

</details>

<details>

<summary>Why is an NVIDIA GPU required?</summary>

At the moment, an NVIDIA graphics card is required for using the ImmersiX application. We are utilising the NVDEC feature which is available on NVIDIA GPUs.

</details>

1. Why does ImmersiX fail to launch or crash immediately?

This issue often arises from insufficient GPU resources or incompatible hardware. Ensure your system meets the minimum requirements:

* **GPU**: GeForce NVIDIA RTX 3080 or better // AMD Ryzen 7 5800X3D or better
* **RAM**: At least 32 GB
* **VRAM**: Minimum 8 GB dedicated
* **OpenXR Runtime**: Properly configured[steamcommunity.com+2steamcommunity.com+2steamcommunity.com+2](https://steamcommunity.com/app/1704450/discussions/0/595139910033217885/?utm_source=chatgpt.com)[steamcommunity.com](https://steamcommunity.com/app/1704450/discussions/0/601891815092660615/?utm_source=chatgpt.com)

\
Additionally, close unnecessary background applications to free up resources. [steamcommunity.com](https://steamcommunity.com/app/1704450/discussions/0/595137806003701918/?utm_source=chatgpt.com)

**Does ImmersiX support AMD GPUs?**

Currently, ImmersiX primarily supports NVIDIA GPUs. However, the developers are actively working on AMD compatibility and have made progress in internal testing. Stay tuned for official updates regarding AMD support.&#x20;

**What does the error "Could not create heap for ring buffer" mean?**

This error indicates a failure in allocating necessary GPU memory, often due to hardware limitations. Users with GPUs below the recommended specifications, such as the GTX 1060, may encounter this issue. Upgrading to a supported GPU is advised.

**How can I resolve the "nvcuda.dll not found" error?**

This error suggests that the application is attempting to access NVIDIA CUDA drivers, which are absent on AMD systems. As ImmersiX currently requires NVIDIA hardware, AMD users may encounter this issue. Await further updates for AMD compatibility. [steamcommunity.com+5steamcommunity.com+5steamcommunity.com+5](https://steamcommunity.com/app/1704450/discussions/0/601891695160802583/?utm_source=chatgpt.com)

**My VR controllers aren't responding in ImmersiX. What should I do?**

Some users have reported that only specific controllers (e.g., the left controller) are functional. Ensure your OpenXR runtime is correctly set, and consider switching to SteamVR if issues persist. Refer to the technical manual for detailed setup instructions.

**How do I set up ImmersiX with my VR headset?**

For Meta Quest users, use the Meta Link OpenXR runtime and enable "Allow unknown sources." For other headsets, ensure the appropriate OpenXR runtime is active. Detailed setup guides are available in the technical manual.

**The application stutters or buffers during playback. How can I improve performance?**

Performance issues can stem from high CPU usage or insufficient system resources. Ensure your hardware meets the recommended specifications, and close unnecessary applications. Adjusting settings in SteamVR or the ImmersiX performance guide may also help.

**Why does the "Construct" video appear pixelated?**

The "Construct" video utilizes an older version of ImmersiX's technology, which may result in lower visual quality. Upcoming content will feature enhanced visuals and improved technology.

**What content is currently available in ImmersiX?**

ImmersiX offers immersive experiences like "Construct VR" and "Weightless" that are completely free experiences and "Sharkarma-Guardians of the Ocean" an ocean adventure movie designed for 6DoF VR storytelling. Additional content is in development and will be released in future updates.

**Where can I find the technical manual and performance guide?**

The official technical manual, including setup instructions and performance tips, is available on the ImmersiX Steam Community page.

Any more questions? Please [Contact us](/technologies/immersix/contact-us).


# Troubleshoot guide

Welcome to the ImmersiX Troubleshooting Guide. This guide aims to help you identify and resolve common issues you might encounter while using the ImmersiX application. Whether you're facing performance issues, initialization errors, or playback difficulties, this guide provides solutions to enhance your playback experience. For further assistance, please contact our support team.

{% hint style="warning" %}
If you are experiencing issues with your headset or its connection to the PC, please refer to the instructions from the headset manufacturer. For Meta Quest devices, you can find more information in the Meta Quest [Help Centre](https://www.meta.com/en-gb/help/quest/).
{% endhint %}

{% hint style="info" %}
Question? We're happy to help! Please [Contact us](/technologies/immersix/contact-us).
{% endhint %}

### Performance issue detected

The application has detected a performance issue which may effect your playback experience. We recommend checking out our [Performance guide](/technologies/immersix/performance-guide) to improve your immersive experience.

<figure><img src="https://1563923258-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F14O9SSDxghEHEuA3ThIX%2Fuploads%2FmL5sUjYZg7Vrp1mexfxa%2Fperformance_issues.png?alt=media&amp;token=964ba2fe-a188-45f3-ae60-043f3558bab5" alt=""><figcaption></figcaption></figure>

### No audio device found: Could not initialize scene audio manager

The application could not find an active audio device. Please plug in an audio device.

### VR initialization error -2: Could not create OpenXR instance

If an OpenXR instance cannot be created when launching the ImmersiX application, the Meta Quest Link application is not running. Please refer to step 3 in [Getting started](/technologies/immersix/getting-started) to install and set up the Meta Quest Link before starting the ImmersiX application.

<figure><img src="https://1563923258-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F14O9SSDxghEHEuA3ThIX%2Fuploads%2FoT1v94sG1AWBDE84SOQC%2FOpenXR%20error%20-2.png?alt=media&amp;token=651302bd-2578-4265-801f-42a9a7e25a4d" alt="Error message stating OpenXR is not initialised."><figcaption><p>OpenXR error (-2)</p></figcaption></figure>

### VR initialization error -4: No OpenXR headset or system found

If no OpenXR headset or system is found when launching the ImmersiX application, the Meta Quest Link application is not set-up correctly. Please refer to steps 4-6 in [Getting started](/technologies/immersix/getting-started) to set up the Meta Quest Link before starting the ImmersiX application.

<figure><img src="https://1563923258-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F14O9SSDxghEHEuA3ThIX%2Fuploads%2FhS0vnhFMHVKTFn80hehH%2FOpenXR%20error%20-4.png?alt=media&amp;token=1959a577-a82f-4ee4-8b94-4066c05df500" alt=""><figcaption><p>OpenXR error (-4)</p></figcaption></figure>

### GPU warning

The ImmersiX app requires an NVIDA graphics card (GPU) with the minimum recommended GPU being an NVIDIA GeForce GTX 1080 Ti (see [System requirements](/technologies/immersix/system-requirements) for more details). If this warning is shown, your system does either not include an NVIDIA GPU or the available GPU does not meet the minimum system requirements. You will likely observe playback issues when using the ImmersiX application.

<figure><img src="https://1563923258-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F14O9SSDxghEHEuA3ThIX%2Fuploads%2Feb5s57bgjMz1AVH7Ndfc%2FGPU_error.png?alt=media&amp;token=bacdca04-5a4a-48cb-9e29-ca73e887c18e" alt=""><figcaption><p>GPU warning</p></figcaption></figure>


# Performance guide

Thank you for watching volumetric movies with ImmersiX. Viewing our 6 degrees-of-freedom immersive movies requires a lot from a computer and sometimes even a computer that meets all minimum performance specifications may struggle to play back the content smoothly. Luckily, a lot of these cases can be resolved with a simple configuration change. In this document we have gathered all the advice and troubleshooting information that has helped our users enjoy volumetric movies without interruptions and in the quality they were meant to be seen.

## General Performance Advice <a href="#general" id="general"></a>

The following advice may improve overall system performance when playing back volumetric movies.

#### Close other software and windows

Other software running on the computer may be using valuable resources that could otherwise be used to play the movie. Notable examples include screen recording software and GPU-accelerated web browsers. For an optimal volumetric movie experience please exit any application that you are not currently using and close or minimize as many windows as possible.

#### Temperature

When the computer's temperature rises through intensive use some components (usually the processor and graphics card) may be slowed down as a protective measure. If you find that volumetric movie playback is not smooth as a result of the system overheating you should try to lower the computer's temperature.

* Make sure the cooling vents are free of dust and other obstructions.
* (Desktop computer) Take off the computer case's side panel for improved airflow.
* (Laptop computer) Use a cooling stand.
* Move the computer to a cooler room or lower the room temperature.

#### Power plan

Make sure that Windows is using the "High performance" power plan (or "Ultimate performance" if it is available) while playing the movie. Other power plans may limit system performance. If the computer is a laptop, make sure that the power adapter is plugged in while playing the movie.

#### Single channel memory&#xD;

If a computer contains only a single system memory (RAM) module, the machine is generally insufficient to play back volumetric movies. Adding more memory to the computer to spread the load over two or more modules in a dual channel or quad channel memory setup will result in much better performance. If you are unsure how many memory modules your computer has, please look at the reported value in the ImmersiX application log (`prz_applog.txt` in the ImmersiX install directory).

## Improving File Reading Performance <a href="#reading" id="reading"></a>

Playing back volumetric movies requires a large amount of data to be read with little tolerance for delays. Our movies have been carefully tuned to be played back smoothly from a regular SSD, but other factors than the drive's read speed may affect playback. When movie playback is pausing frequently because of file reading issues the advice in this section may help.

#### Storage drive connection

We expect the SSD to be connected to the motherboard with SATA 3 (also known as 'SATA III', 'SATA 600' or 'SATA 6 Gbit/s') or better (eg. M.2 or PCIe). External enclosures or network drives may also work but if you are unsure whether their performance is sufficient please try installing the movie on an internal drive.

#### Anti-virus software

On-access threat scanning by anti-virus software may add significant delay to file reading operations. Sometimes when this happens it is enough to let the player play through the entire movie so that all files are accessed at least once, and subsequent playthroughs will be smooth. If the anti-virus software continues to interfere with movie playback it is usually possible to temporarily pause on-access scanning or exclude the folder containing the movie files from scanning through the anti-virus software user interface.

#### General advice

It is useful to also read the [#general](#general "mention") section as the issues and solutions described there may affect file reading performance.

## Improving Movie Decompression Performance <a href="#decoding" id="decoding"></a>

Our volumetric movies make extensive use of compression to keep the storage space and file reading bandwidth requirements reasonable. Decompressing the movie during playback requires a sufficiently fast processor and DDR4 or faster system memory. When movie playback is pausing frequently because of decompression issues the advice in this section may help.

#### Core parking

Core parking is a power-saving measure available on modern processors. When core parking is enabled you may experience short processor stalls when cores are transitioning from parked to non-parked state. These stalls can be severe enough to disrupt the smooth playback of a volumentric movie. Disabling core parking requires changes to power plan configuration and other settings, or you can use a 3rd party tool to achieve this.

#### General advice

It is useful to also read the [#general](#general "mention") section as the issues and solutions described there may affect movie decompression performance.

## Improving Transfer to GPU Memory <a href="#uploading" id="uploading"></a>

In some cases, moving the volumetric movie data from system memory into the graphics card memory becomes a bottleneck preventing smooth playback of the movie. This usually happens when the graphics card is throttled down to prevent overheating or when the computer's power plan prevents full use of the PCIe lanes for data transfer to the graphics card. In these situations the advice in the General Performance Advice section may help.

Rarely, it also happens that the graphics card does not have access to the full amount of PCIe lanes that it normally would have. Try moving any PCIe or M.2 expansion cards or SSDs to a different slot on the motherboard, or removing them altogether. You may need to change some UEFI (BIOS) settings to resolve this issue.

## Improving 3D Rendering Performance <a href="#rendering" id="rendering"></a>

A steady framerate is key to an enjoyable virtual reality experience. When you are experiencing regular hitches or a too low framerate the advice in this section may help.

#### Too many VR runtimes

Since the immersiX player is a native OpenXR application, it can interact directly with any VR runtime that supports OpenXR. A situation that arises often is users with Meta Quest headsets launching the immersiX player through SteamVR. The SteamVR runtime will then act as an intermediary between the player and the Meta Quest runtime, adding an unnecessary layer of indirection. To resolve this, open the Meta Quest Link application settings and set the active OpenXR runtime to Meta Quest Link.

#### Disconnect secondary displays

Having multiple displays connected to the computer means that the graphics card has to spend some effort to keep all of them updated. Disconnecting any displays that you are not using while watching the movie is an easy way to free up some rendering resources.

#### NVIDIA power setting

Your NVIDIA graphics card can be set to use 'Normal' or 'Maximum performance' power management mode. This setting can be changed in the NVIDIA Control Panel under 'Manage 3D settings'.

#### Lower VR rendering resolution

Most VR runtimes allow you to change the VR rendering resolution in their settings, making it possible to lower the load on the graphics card at the cost of visual quality. This trade-off is often worth it to achieve a stable framerate, so adjust this setting to your personal preference.

#### General advice

It is useful to also read the [#general](#general "mention") section as the issues and solutions described there may affect rendering performance.

{% hint style="info" %}
If you continue to experience performance issues despite your device meeting the [System requirements](/technologies/immersix/system-requirements), you can [Contact us](/technologies/immersix/contact-us).
{% endhint %}


# Contact us

This page provides information on how to reach out to us if your queries remain unresolved after consulting our [Frequently asked questions](/technologies/immersix/frequently-asked-questions) or [Troubleshoot guide](/technologies/immersix/troubleshoot-guide). Our team is here to assist you with any further questions or concerns you might have. For prompt assistance, please use the contact options listed below to ensure your inquiries are addressed effectively.

{% stepper %}
{% step %}

### :desktop: Technical enquiry

If you are having troubles running ImmersiX on your machine, please have a look at the steps outlined in the [Troubleshoot guide](/technologies/immersix/troubleshoot-guide) and [Performance guide](/technologies/immersix/performance-guide). If, after following these instructions, you continue to have issues, you can contact us.

To get support with a technical enquiry, please send an email to <immersix-support@v-nova.com>.

When contacting us, please ensure to include the following information:

1. Steam build ID
2. System information: CPU, GPU, RAM and SSD type (e.g. SATA III, NVMe)
3. Type of VR headset
4. Content & timestamp where you are experiencing issues
   {% endstep %}

{% step %}

### :shopping\_cart: Purchase enquiry

For any enquiries related to a purchase on Steam, please contact the [Steam Support](https://help.steampowered.com/).
{% endstep %}

{% step %}

### :e-mail: General enquiry

For any other topics, you can contact us via email at <immersix-support@v-nova.com>.
{% endstep %}
{% endstepper %}


# Privacy Policy for Steam

Privacy Policy for ImmersiX (Last Updated: 31 March 2025)

## 1. Introduction

Welcome to ImmersiX! This Privacy Policy explains how we collect, use, and protect your data when using our immersive video application on Steam.

By using ImmersiX, you agree to the terms outlined in this policy. If you do not agree, please discontinue use of the app.

## 2. Who We Are

* Company Name: V-Nova Limited
* Registered Office: 20 Eastbourne Terrace, Paddington, London, England, W2 6LG
* EU Subsidiary: PARALLAXTER srl, Rue Deschampheleer 26/17 – 1081 Koekelberg, Belgium
* Contact Email: <legal@v-nova.com>

For EU users, our EU subsidiary serves as the primary contact for data protection matters under the EU GDPR.

## 3. Data We Collect

We collect pseudo-anonymized analytics data to improve our app’s performance and user experience. The types of data collected may include:

* Usage Data: Time spent in-app, video playback interactions, settings adjustments.
* Device Information: Hardware information, VR headset model, randomised ID per installation.

**We do NOT collect** personal identifiers such as names, email addresses, precise location, or financial/payment information.

## 4. How We Use Your Data&#x20;

We use the collected data to:

* Improve app performance and user experience.
* Analyse user engagement for future feature development.
* Detect and resolve technical issues.

We do not sell or share your data with third parties for marketing purposes.

## 5. Legal Basis for Processing

Under UK GDPR & EU GDPR, we process analytics data based on legitimate interest (to improve ImmersiX) or user consent where required.

## 6. How Long We Keep Your Data

We retain analytics data for 12 months, after which it is automatically deleted or anonymized.

## 7.  Your Rights (For UK/EU Users)

As a user, you have the right to:

* Opt-out of analytics tracking (via in-app settings).
* Request access to your data.
* Request deletion of your identifiable data (if applicable).
* Lodge a complaint with the UK’s ICO or your EU country’s Data Protection Authority if you believe your rights are violated.

## 8. Opt-Out of Analytics

ImmersiX includes an in-app setting to disable analytics tracking. Users can disable data collection at any time by toggling the analytics opt-out option found in the settings menu. Once disabled, your usage data will no longer be collected.

## 9. Data Storage & Transfers

We store analytics data securely on AWS servers located in Ireland (EU). Data may be accessed by authorized personnel in the United Kingdom for operational and analytical purposes. All access is governed by strict internal security protocols and complies with applicable data protection laws.

## 10. Third-Party Services

We do not use Google Analytics. All analytics data is collected and processed using our internal tools. Steam may collect its own analytics data independent of ImmersiX. Please refer to Steam’s Privacy Policy for more information on how Valve Corporation processes user data.

## 11. Data Sharing Within Our Group

We may share analytics data collected through ImmersiX with other companies within our corporate group (our “affiliates”) where necessary for legitimate internal business purposes, such as app performance monitoring, troubleshooting, or product development. All such sharing is limited to pseudo-anonymized data and is governed by appropriate internal safeguards and confidentiality agreements.

Our affiliates are subject to this Privacy Policy and only process data in accordance with applicable data protection laws.

## 12. Contact Us

If you have any questions or privacy concerns, please contact us at <legal@v-nova.com>.

For EU users, please direct GDPR-related inquiries to our EU subsidiary at <legal@v-nova.com>




---

[Next Page](https://docs.v-nova.com/technologies/llms-full.txt/1)