Decoder Integration Layer (DIL)
Introduction
The DIL (Decoder Integration Layer) is the V-Nova library used to integrate LCEVC decoding and rendering to an existing legacy decoder implementation.
The DIL can be used in both:
Application integration - the DIL renders directly onto an on-screen surface, provided by the client, of arbitrary size (generally different from the content resolution). Examples: ExoPlayer (Android).
OS Integration - the DIL decodes onto a buffer or draws onto an off screen texture of the same size of the content final resolution, it doesn’t handle the final render to display, including YUV to RGB conversion, and resizing to the destination surface. Examples: MFT (Windows), OMX (Android).
The following block diagram illustrates the role of the DIL more clearly. In conjunction with a base decoder, typically provided by the OS, it offers a complete solution from buffer to output.
Functionality
Input & output
As shown in the diagram above, the DIL needs two inputs:
input buffers - the same buffers that are fed to the base decoder in encoding order;
base decoded frames - produced by the base decoder in presentation order;
and produces an output onto either a
buffer or
offscreen texture or
onscreen surface.
Colour formats
The DIL can operate on a base decoded picture in the following colour formats:
YUV420 I420 - planar Y, U and V with 4:2:0 subsampled chroma;
YUV420 NV12 - semi-planar Y (planar), and UV (interleaved) with 4:2:0 subsampled chroma;
RGB/RGBA - raster Red Green Blue, i.e. on one plane with interleaved colour components;
Modes
The DIL can be configured to work with two types of internal pipeline:
CPU - all the LCEVC stages are performed in CPU, only using SIMD acceleration, by means of the DPI only, the GPU is used only for the possible YUV/RGB conversion;
GPU - most of the LCEVC stages are performed in GPU suing GL shaders, including YUV/RGB conversions, while the CPU is only used to produce the LCEVC residual planes at both LOQs
Output type
The DIL offers decoding to either
on screen
off screen
On screen refers to a window from the OS window system, currently EGL and GLFW windowing systems are supported.
Off screen refers to an array of buffers or GL textures in memory. The latter case has been designed to allow the client to manage decode and presentation separately, for example to allow buffering/queuing already decoded pictures before display.
Other configurable features
On creating a DIL instance, the client can configure additional features, such as:
OpenGL major and minor versions (to force a specific version over the system’s selected one);
Use of OpenGL ES;
Use of 8 bit LCEVC residual planes - instead of 16 bit
Use of Hardware Buffers - (Apple and Android only)
Enable an on screen UI for stats and live config;
Enable dumping stats to local storage;
Enable dumping raw output frames to local storage;
The clients configures the DIL at creation time by passing a JSON string and a pair of window system related parameters for context and destination surface.
API Overview
Using the DIL is fairly easy, especially if compared to the more basic DPI, since it operates at a relatively high abstraction level. Apart from obvious creation and destruction of a DIL instance (respectively DIL_Create()
and DIL_Destroy()
), the main API calls are effectively two:
Feed input -
DIL_AddNALData()
Decode -
DIL_DecodeandRender()
(for decoding on screen, alternativelyDIL_Decode()
for decoding off screen followed byDIL_Render()
)Feed input -
DIL_AddNALData()
Decode -
DIL_DecodeandRender()
(for decoding on screen, alternativelyDIL_Decode()
for decoding off screen followed byDIL_Render()
)
The DIL also offers an API for the client to retrieve the size of the output picture, which can be useful to pre-allocate buffers or textures of the right size before decoding:
Get size information -
DIL_GetDecodeInformation()
Note: input buffers must have NAL byte stream format with either Annex B or Length Prefix format.
Drop/Skip frames
LCEVC content may be encoded with a temporal feature that requires each and every frame to be actioned on in order to keep the DPI internal temporal reference data correct. As a result when the client skips frames, for ex. because of a seek in the timeline, or drops frames because they are “late”, it shall let the DIL know by calling the following function:
DIL_DecodeSkip()
The DIL does know whether the temporal feature is on and therefore can fall back to a no operation case if that is the case. It is not advised that the client tries to implement this behaviour on its side.
Synchronous and Asynchronous modes
The DIL can be used in both synchronous or asynchronous mode. This is applicable to the decode and render functions:
DIL_Decode()
DIL_DecodeAndRender()
DIL_DecodeSkip()
DIL_Render()
If the client has previously set the callback function by means of the DIL_SetDecodedAsyncCallback()
or DIL_SetRenderAsyncCallback()
API calls, the above decode functions will work asynchronously, therefore returning immediately and triggering the client’s callback function when the decode (or render) has completed. If the callback has not been set the functions will work synchronously.
Base pass-through mode
The DIL also supports a pass-through mode, in which the base is simply copied on to the output without applying the LCEVC enhancement. This mode is triggered when no LCEVC data is found for the frame and can be set as forced behaviour, for every frame, from the JSON configuration.
Example Integration Code
In basic terms, a sample integration code using the DIL can be briefly described, among the various phases, as follows:
DIL API
DIL_Create
Create an instance of a DIL Decoder
json_settings JSON string with initialisation parameters
context_settings settings from the windowing system
instance decoder instance created
return DIL API return code
DIL_Destroy
Destroy an instance of a DIL Decoder
decoder instance to be destroyed
DIL_GetImage
Returns a fresh usable instance of DIL_Image
decoder DIL instance
image_desc Describing Image configurations
image Result DIL_Image if creation is successful, NULL otherwise
return DIL_RC_Error if image_desc contains incompatible values
DIL_ReleaseImage
Releases instance of DIL_Image
NOTE: No reference calls to image address should be made after this method
decoder DIL instance
image DIL_Image instance to be released
return DIL API return code
DIL_ImageGetDesc
Will return DIL_ImageDesc of the image
image DIL_Image to query
desc Contents of the pointer will be rewritten with image descriptions
return DIL API return code
DIL_ImageSetActiveRegion
Will set the active region for an image
image DIL_Image instance to modify
offset_x X offset for the region
offset_y Y offset for the region
width width for the region
height height for the region
return DIL_RC_Error if crop region is out of bounds, image_type doesn't support cropping
DIL_ImageGetActiveRegion
Will get the active region for an image
image DIL_Image instance to modify
offset_x X offset for the region
offset_y Y offset for the region
width width for the region
height height for the region
return DIL_RC_Error if crop region is out of bounds, image_type doesn't support cropping
DIL_ImageGetPlanesCount
Will return number image's number of planes
image DIL_Image to query
plane_count Will be filled with number of planes
return DIL API return code
DIL_ImageSetPlaneBuffer
Will set content and configurations of a plane by data buffer
image DIL_Image instance to modify
plane_index Index of the plane to modify
buffer_desc DIL_ImagePlaneBufferDesc instance to read from
return DIL_RC_Error if plane_index is out of bounds, image_type is not DIL_Buffer or image is managed
DIL_ImageGetPlaneBuffer
Will return content and configurations of a plane
image DIL_Image to query
plane_index Index of the plane to query
buffer_desc DIL_ImagePlaneBufferDesc instance to write to
return DIL_RC_Error if plane_index is out of bounds or image_type is not DIL_Buffer
DIL_ImageSetPlaneTexture
Will set content and configurations of a plane by OpenGL/GLES texture
image DIL_Image instance to modify
plane_index Index of the plane to modify
texture_desc DIL_ImagePlaneTextureDesc instance to read from
return DIL_RC_Error if plane_index is out of bounds, image_type is not DIL_Texture or image is managed
DIL_ImageGetPlaneTexture
Will return content and configurations of a plane by OpenGL/GLES texture
image DIL_Image instance to modify
plane_index Index of the plane to modify
texture_desc DIL_ImagePlaneTextureDesc instance to read from
return DIL_RC_Error if plane_index is out of bounds, image_type is not DIL_Texture or image is managed
DIL_ImageSetPlaneHardwareBuffer
Will set content and configurations of a plane by OS specific Hardware buffer.
image DIL_Image instance to modify
plane_index Index of the plane to modify
desc DIL_ImagePlaneHardwareBufferDesc instance to read from
return DIL_RC_Error if plane_index is out of bounds, image_type is not DIL_Texture or image is managed
DIL_ImageGetPlaneHardwareBuffer
Will return content and configurations of a plane by OS specific Hardware buffer.
image DIL_Image instance to query
plane_index Index of the plane to query
desc DIL_ImagePlaneHardwareBufferDesc instance to write to
return Will return DIL_RC_Error if plane_index is out of bounds or image_type is not DIL_HardwareBuffer
DIL_ImageSetUserData
Will set the user data associated with the image
image DIL_Image to set the userdata for
userdata The userdata to associate with the image
return DIL_RC_InvalidParam if the image, DIL_RC_Success otherwise
DIL_ImageGetUserData
Will get the user data associated with the image through DIL_ImageSetUserData
image DIL_Image to get the userdata for
userdata The userdata to associate with the image
return DIL_RC_InvalidParam if the image is invalid or userdata is invalid, DIL_RC_Success otherwise
DIL_ImagePlaneHardwareBufferLock
Will return content and configurations of a plane buffer obtained by locking the hardware buffer for CPU access
image DIL_Image to access
plane_index Index of the plane to query
read_access Whether we want read access
write_access Whether we want write access
buffer_desc DIL_ImagePlaneBufferDesc instance to write to
return Will return DIL_RC_Error if plane_index is out of bounds or image_type is not DIL_HardwareBuffer
DIL_ImagePlaneHardwareBufferUnlock
Will unlock the hardware buffer plane once CPU access is completed.
image DIL_Image to unlock
plane_index Index of the plane to query
return Returns DIL_RC_Error if plane_index is out of bounds or image_type is not DIL_HardwareBuffer
DIL_AddTSData
Feed Transport Stream packet(s) for the DIL to extract and store LCEVC data, feed a buffer of TS packets associated with the Access Unit identified by input cc and timestamp
NOTE: input_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback
NOTE: input_cc shall have monotonically increasing value over calls
decoder DIL instance
input_cc input continuity counter
data pointer to the TS buffer
size size of the TS buffer
returns DIL API return code
DIL_AddWebmData
Feed Webm payload for the DIL to extract and store LCEVC data
decoder DIL instance
input_cc input continuity counter
data pointer to the TS buffer
size size of the TS buffer
returns DIL API return code
DIL_AddNALData
Feed NAL unit(s) for the DIL to extract and store LCEVC data, feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp
NOTE: input_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback
NOTE: input_cc shall have monotonically increasing value over calls
NOTE: timestamp shall have monotonically increasing value over calls within the same input_cc
decoder DIL instance
input_cc input continuity counter
timestamp time reference for the passed NAL units
data pointer to the NAL units buffer
size size of the NAL units buffer
nalu_format format of the NAL units
return DIL API return code
DIL_AddNALDataEx
Feed NAL unit(s) for the DIL to extract and store LCEVC data (Extended), feed a buffer of NAL units associated with the Access Unit identified by input cc and timestamp and additionally let the DIL strip the LCEVC payloads and zero pad the trailing bytes, returning the new size before padding
NOTE: input_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback
NOTE: input_cc shall have monotonically increasing value over calls
NOTE: timestamp shall have monotonically increasing value over calls within the same input_cc
decoder DIL instance
input_cc input continuity counter
timestamp time reference for the passed NAL units
data pointer to the NAL units buffer
size size of the NAL units buffer
nalu_format format of the NAL units
return size of the stripped NAL units buffer
DIL_AddRawData
Feed LCEVC data to the DIL, feed a buffer of pre parsed LCEVC payload data for the Access Unit identified by input cc and timestamp
NOTE: input_cc is a feed counter identifying which feed the passed data belongs to, the client shall increment its value every time the input changes, typically because of a playback position change or a rendition change in an ABR playback
NOTE: input_cc shall have monotonically increasing value over calls
NOTE: timestamp shall have monotonically increasing value over calls within the same input_cc
decoder DIL instance
input_cc input continuity counter
timestamp time reference for the passed LCEVC data
data pointer to the NAL units buffer
size size of the NAL units buffer
return DIL API return code
DIL_GetDecodeInformation
Get details about the decoding process for the Access Unit identified by input cc and timestamp
decoder DIL instance
input_cc input continuity counter
timestamp time reference of the picture
base_width width of the base decoded picture
base_height height of the base decoded picture
information pointer to decoder information structure that the DIL will fill
return DIL API return code
DIL_Decode
Decode on to an off screen DIL_Image, get an off screen LCEVC enhanced (or base pass-through) picture from the input base picture relative to input cc and timestamp
decoder DIL instance
input_cc input continuity counter
timestamp time reference of the picture to be decoded
max_decode_time_us maximum amount of time the decode has to complete, use 0 to disable decode timeout
input pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)
output pointer to LCEVC enhanced (or base pass-through) output picture
decode_information pointer to decoder information structure that the DIL will fill (if DIL_SetDecodedAsyncCallback has not been called)
return DIL API return code
DIL_Render
Render to back buffer a previously decoded DIL_Image
NOTE: for the scheduled render to be set DIL_SetRenderAsyncCallback must have been previously called
decoder DIL instance
input_cc input continuity counter
timestamp time reference of the picture to be rendered
image pointer to a picture to be rendered to back buffer
render_information pointer to a render information structure
delay_us delay in microseconds for the render to eb executed (0 for no delay)
return DIL API return code
DIL_DecodeAndRender
Decode and render immediately to back buffer
decoder DIL instance
input_cc input continuity counter
timestamp time reference of the picture to be decoded and rendered
max_decode_time_us maximum amount of time the decode has to complete, use 0 to disable decode timeout
input pointer to decoded base picture for input cc and timestamp (it may be modified if DIL operates in CPU mode)
render_information pointer to a render information structure
decode_information pointer to decoder information structure that the DIL will fill (if DIL_SetDecodedAsyncCallback has not been called)
return DIL API return code
DIL_DecodeSkip
Let the DIL know that an Access Unit is not being presented (dropped or skipped), DIL will do the minimum processing to keep internal state consistent for the next picture
NOTE: this call will generate a DecodeAsyncCallback if previously set
NOTE: calling this on an IDR Access Unit is undefined behaviour
decoder DIL instance
input_cc input continuity counter
timestamp time reference for the Access Unit to be skipped
return DIL API return code
DIL_Flush
Synchronize client and DIL by throwing away any pending decodes and renders and waiting for all callbacks to complete
NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called with a the result filed in the callback will indicate the error DIL_RC_Flushed making DIL calls in this callback could result in a deadlock use the DIL_SetFlushAsyncCallback to break this loop
decoder DIL instance
return DIL API return code
DIL_Drain
synchronize client and DIL by allowing any pending decodes and renders to complete and waiting for all callbacks to complete
NOTE: all callbacks set by DecodeAsyncCallback and RenderAsyncCallback will be called making DIL calls in this callback could result in a deadlock use the DIL_SetDrainAsyncCallback to break this loop
decoder DIL instance
return DIL API return code
DIL_GetGLContext
Get the DIL's GL Context
decoder DIL instance
context pointer to pointer of GL Context
return DIL API return code
DIL_GetGLDisplay
Get the DIL's GL Display
decoder DIL instance
display pointer to pointer of GL Display
return DIL API return code
DIL_SetGLWindow
Set the DIL's GL output window
decoder DIL instance
gl_external_window pointer to destination window
is_secure true if the destination window is secure
return DIL API return code
DIL_SetExitRequestedCallback
Set ExitRequest call back function
decoder DIL instance
callback pointer to DILOnExitRequestedCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_SetDecodedAsyncCallback
Set asynchronous on decoded call back function
result result of the decode request, see DIL_ReturnCode
input_cc input continuity counter
timestamp time reference of the picture that was decoded
input pointer to base input picture, the same passed in the previous DIL_Decode
output pointer to LCEVC enhanced (or base pass-through) output picture, the same passed in the previous DIL_Decode
information pointer to decoder information structure (not the same passed in from the previous DIL_Decode)
user_data pointer to the same client's user data that was passed into the DIL_SetDecodedAsyncCallback
decoder DIL instance
callback pointer to DILOnDecodedAsyncCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_SetRenderAsyncCallback
Set asynchronous on render callback function
result result of render request, see DIL_ReturnCode
completion_time_us actual time in microseconds the render completed
input_cc input continuity counter
timestamp time reference of the picture that was rendered
output pointer to the same picture passed in with the DIL_Render method
user_data pointer to the same client's user data that was passed into the DIL_SetRenderAsyncCallback
decoder DIL instance
callback pointer to DILOnRenderAsyncCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_SetRenderResizeAsyncCallback
Called at the start of the render process when the image to be displayed has a different size to the last image rendered. This is a blocking call and there should be no DIL calls in the callback function.
result result of render request, see DIL_ReturnCode
completion_time_us actual time in microseconds the render completed
input_cc input continuity counter
timestamp time reference of the picture that was rendered
output pointer to the same picture passed in with the DIL_Render method
user_data pointer to the same client's user data that was passed into the DIL_SetRenderAsyncCallback
decoder DIL instance
callback pointer to DILOnRenderAsyncCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_SetFlushAsyncCallback
Callback function for the DIL_Flush() function
user_data pointer to client's user data
decoder DIL instance
callback pointer to DILOnSynchronizeCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_SetDrainAsyncCallback
Callback function for the DIL_Drain() function
user_data pointer to client's user data
decoder DIL instance
callback pointer to DILOnSynchronizeCallBack function
user_data pointer to client's user data
return DIL API return code
DIL_InputMouseButtonState
Pass the DIL the position of the input mouse button
decoder DIL instance
pressed true if button is pressed
return DIL API return code
DIL_InputMousePosition
Pass the DIL the position of the input mouse
decoder DIL instance
x horizontal coordinate of the mouse cursor
y vertical coordinate of the mouse cursor
return DIL API return code
DIL Types and Enumerations
DIL_NALFormat enum
DIL_NALFormat represents the two possible NAL Unit formats, for each of the MPEG base codecs the DIL currently supports: H264 (ISO/IEC 14496-10, aka AVC) and H265 (ISO/IEC 23008-2, aka HEVC) NAL, where Annex B indicates 3 byte 0x000001 or 4 byte 0x00000001 start code prefix as described in Annex B of (H264 or HEVC), LP (Length Prefix) indicates a 4 byte prefix with the length of the NAL unit as MSBF 32 bit unsigned int.
DIL_ReturnCode enum
DIL_ReturnCode represents the possible API return codes. Most of the API calls will return one of these values.
DIL_ColorFormat enum
This enum represents the supported colour formats, currently only 8 bit per sample formats are supported For a detailed description of the formats see: https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html
DIL_ImageType enum
This enum identifies the type of planes the DIL_Image consists of; planes are non-overlapping sets of samples belonging to one or more colour channels with a constant sample stride, so RGB is a single plane, because R G and B pointed at separately would overlap UV from NV12 is a single plane, because U and V pointed at separately would overlap Y U and V from I420 are three planes because they do not overlap
DIL_ImageDesc struct
Holds configurations of a single DIL_Image instance
DIL_ContextSettings struct
This structure holds parameters related to the window surface the DIL would render into, in case of on screen rendering those parameters are normally provided by the windowing system, e.g. EGL, GLFW.
DIL_RenderInformation struct
This structure captures the properties related to the (on screen) window rendering, they do not apply for buffer or texture output (off screen). NOTE: rotation is a per frame property because it may change frame by frame due to for ex. the encoding device rotating while shooting.
DIL_DecodeInformation struct
This structure captures properties related to the decoding process, width and height of the output, whether LCEVC data has been found and whether it has been applied.
DIL_ImagePlaneBufferDesc struct
This struct describes a plane memory buffer
DIL_ImagePlaneTextureDesc struct
This struct describes a plane OpenGL texture
DIL_HardwareBufferUsageOption enum
This enum use used to set the UsageOption when creating HardwareBuffer DIL_Image types. It should be provided in the "hardware_buffer_usage_option" tag when configuring the DIL object
DIL_ImagePlaneHardwareBufferDesc struct
This struct describes a plane HardwareBuffer
DIL Properties API
DIL_QueryProperty
Obtain a specific named property. This function provides a way of looking up a property by name to observe it's current value, this may be of use for a user to query DIL properties
DIL_SetProperty
Set a specific named property. This function provides a way of changing a property by name
DIL_QueryPropertyGroup
Obtain all the available properties for a named group
DIL_ReleaseProperty
Release a previously queried property, this function provides a way of releasing any memory allocated by a call to DIL_QueryProperty. Any data contained in value maybe invalidated after this call
DIL_ReleasePropertyGroup
Release a previously queried property group, this function provides a way of releasing any memory allocated by a call to DIL_QueryPropertyGroup Any data contained in value maybe invalidated after this call
DILProperty struct.
This structure contains all relevant information used to query a property or metadata from the DIL.
DILPropertyGroup struct.
This structure contains a logical grouping of properties, this is a convenience feature to assist with providing a "pretty" command line, or GUI.
All properties are assigned to a group, and properties can only appear in one group.
Last updated