Decoder Integration Layer (DIL)

Introduction

The DIL (Decoder Integration Layer) is the V-Nova library used to integrate LCEVC decoding and rendering to an existing legacy decoder implementation.

The DIL can be used in both:

• Application integration - the DIL renders directly onto an on-screen surface, provided by the client, of arbitrary size (generally different from the content resolution). Examples: ExoPlayer (Android).

• OS Integration - the DIL decodes onto a buffer or draws onto an off screen texture of the same size of the content final resolution, it doesn’t handle the final render to display, including YUV to RGB conversion, and resizing to the destination surface. Examples: MFT (Windows), OMX (Android).

The following block diagram illustrates the role of the DIL more clearly. In conjunction with a base decoder, typically provided by the OS, it offers a complete solution from buffer to output.

Figure 1 Unmodified and LCEVC Pipeline using the DIL

DIL Types and Enumerations

DIL_NALFormat enum

DIL_NALFormat represents the two possible NAL Unit formats, for each of the MPEG base codecs the DIL currently supports: H264 (ISO/IEC 14496-10, aka AVC) and H265 (ISO/IEC 23008-2, aka HEVC) NAL, where Annex B indicates 3 byte 0x000001 or 4 byte 0x00000001 start code prefix as described in Annex B of (H264 or HEVC), LP (Length Prefix) indicates a 4 byte prefix with the length of the NAL unit as MSBF 32 bit unsigned int.

typedef enum DIL_NALFormat
{
    DIL_UnknownNALFormat = 0,
    DIL_H264_AnnexB,
    DIL_H265_AnnexB,
    DIL_H264_LP,
    DIL_H265_LP
} DIL_NALFormat;

DIL_ReturnCode enum

DIL_ReturnCode represents the possible API return codes. Most of the API calls will return one of these values.

typedef enum DIL_ReturnCode
{
    DIL_RC_Success      =  0,       /* The API call completed successfully */
    DIL_RC_Error        = -1,       /* A generic catch all error */
    DIL_RC_InvalidParam = -2,       /* The user supplied an invalid parameter to the function call */
    DIL_RC_NotFound     = -3,       /* The query call failed to find the item by name, but didn't not generate an error. */
    DIL_RC_NotSupported = -4,       /* The functionality requested is not supported on the running system. */
    DIL_RC_Flushed      = -5,       /* The requested operation failed because it was flushed via DIL_Flush. */
    DIL_RC_Timeout      = -6        /* The requested operation failed because it timedout. */
} DIL_ReturnCode;

DIL_ColorFormat enum

This enum represents the supported colour formats, currently only 8 bit per sample formats are supported For a detailed description of the formats see: https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html

typedef enum DIL_ColorFormat
{
    DIL_UnknownColorFormat = 0,     /* Unspecified colour format */
    DIL_I420_8,                     /* 8 bit 4:2:0 YUV Planar colour format: Y, U and V on separate planes */
    DIL_NV12_8,                     /* 8 bit 4:2:0 YUV SemiPlanar colour format: Y plane and UV interleaved plane */
    DIL_RGB_8,                      /* 8 bit Interleaved R, G, B planes 24 bit per sample */
    DIL_RGBA_8,                     /* 8 bit Interleaved R, G, B and A planes 32 bit per sample */
    DIL_I420_10LE,                  /* 10 bit Little Endian 4:2:0 YUV Planar colour format: Y, U and V on separate planes */

} DIL_ColorFormat;

DIL_ImageType enum

This enum identifies the type of planes the DIL_Image consists of; planes are non-overlapping sets of samples belonging to one or more colour channels with a constant sample stride, so RGB is a single plane, because R G and B pointed at separately would overlap UV from NV12 is a single plane, because U and V pointed at separately would overlap Y U and V from I420 are three planes because they do not overlap

typedef enum DIL_ImageType
{
    DIL_UnknownImageType = 0,       /* Unspecified image type */
    DIL_Buffer_Array,               /* One buffer per plane, even if image is on a single memory allocation, an array of planes shall be set by assigning the right pointers to plane buffers. DIL allocated buffers have contiguous memory allocation */
    DIL_Texture_Array,              /* One texture per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
    DIL_HardwareBuffer_Array        /* One HardwareBuffer per plane, typically in YUV colour formats, it consists of a single plane, even for YUV, if GL YUV extension is used */
} DIL_ImageType;

DIL_ImageDesc struct

Holds configurations of a single DIL_Image instance

typedef struct DIL_ImageDesc
{
    bool            is_managed;     /* True means that Image (and its memory) is managed by DIL  */
    DIL_ImageType   image_type;     /* Whether it is a buffer array or a texture array */
    DIL_ColorFormat color_format;   /* Colour format */
	  uint32_t        width;          /* Nominal net width of the image in luma samples, i.e. no alignment to macroblocks or striding */
    uint32_t        height;         /* Nominal net height of the image in luma samples, i.e. no sliceheight or padding */
    bool            can_modify;     /* Specifying wether content of image is readOnly or changable */
    bool            can_resize;     /* Specifying wether allocated memory is resizable or not. (Does not apply to DIL_Buffer_External) */
    uint32_t        options;        /* options used in creating the Image, in the case of HardwareBuffers this is a combination of  DIL_HardwareBufferUsageOption */
} DIL_ImageDesc;

DIL_ContextSettings struct

This structure holds parameters related to the window surface the DIL would render into, in case of on screen rendering those parameters are normally provided by the windowing system, e.g. EGL, GLFW.

typedef struct DIL_ContextSettings
{
    void* gl_external_window;       /* Surface window, from the windowing system */
    void* gl_external_context;      /* Context, from the windowing system */
    bool  is_secure;                /* True if the client has initialised window and context as secure for protected content */
} DIL_ContextSettings;

DIL_RenderInformation struct

This structure captures the properties related to the (on screen) window rendering, they do not apply for buffer or texture output (off screen). NOTE: rotation is a per frame property because it may change frame by frame due to for ex. the encoding device rotating while shooting.

typedef struct DIL_RenderInformation
{
    uint32_t rotation;                  /* Degrees of rotation for the final rendering on the destination window, in 90 unit increments clockwise */
    float    pixel_aspect_ratio_num;    /* Pixel Aspect Ratio of width to height, this is the numerator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
    float    pixel_aspect_ratio_den;    /* Pixel Aspect Ratio of width to height, this is the denominator, par = (pixel_aspect_ratio_num)/(pixel_aspect_ratio_den) */
} DIL_RenderInformation;

DIL_DecodeInformation struct

This structure captures properties related to the decoding process, width and height of the output, whether LCEVC data has been found and whether it has been applied.

typedef struct DIL_DecodeInformation
{
    uint32_t width;                     /* width of the decoded frame */
    uint32_t height;                    /* height of the decoded frame */
    bool     skip_requested;            /* decode skip was requested */
    bool     lcevc_enhanced;            /* true if the frame has been enhanced by lcevc decoding */
    bool     lcevc_available;           /* true if lcevc encoded data is available for this frame */
} DIL_DecodeInformation;

DIL_ImagePlaneBufferDesc struct

This struct describes a plane memory buffer

typedef struct DIL_ImagePlaneBufferDesc
{
    uint8_t* data;                      /* Pointer to first byte of the first active sample of the plane buffer, NOTE: active means no cropping, the active origin is 0,0 */
    uint32_t sample_byte_stride;        /* Distance in bytes between consecutive sample values of the same channel, for ex. UV plane on 8bit depth this will be 2 */
    uint32_t row_byte_stride;           /* Distance in bytes between the first sample of two consecutive rows */
} DIL_ImagePlaneBufferDesc;

DIL_ImagePlaneTextureDesc struct

This struct describes a plane OpenGL texture

typedef struct DIL_ImagePlaneTextureDesc
{
    uint32_t target;                /* OpenGL/GLES texture target, as in glBindTexture target */
    uint32_t id;                    /* OpenGL/GLES texture id. Unused indexes should be set to 0 */
    uint32_t width;                 /* OpenGL/GLES texture width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* OpenGL/GLES texture height, which might have extra padding, NOTE: the active height is in the parent Image struct */
    int32_t  internal_format;       /* OpenGL/GLES (GLint) sized internal format as from Table 2 of glTexImage2D man page */
} DIL_ImagePlaneTextureDesc;

DIL_HardwareBufferUsageOption enum

This enum use used to set the UsageOption when creating HardwareBuffer DIL_Image types. It should be provided in the "hardware_buffer_usage_option" tag when configuring the DIL object

typedef enum DIL_HardwareBufferUsageOption
{
    UsageOption_None = 0,

    UsageOption_Read_Never  = (1 << 0),
    UsageOption_Read_Rarely = (1 << 1),
    UsageOption_Read_Often  = (1 << 2),

    UsageOption_Write_Never  = (1 << 4),
    UsageOption_Write_Rarely = (1 << 5),
    UsageOption_Write_Often  = (1 << 6),

    UsageOption_Protected_Content = (1 << 7),

    UsageOption_GPU_Read          = (1 << 8),
    UsageOption_GPU_Write         = (1 << 9)
} DIL_HardwareBufferUsageOption;

DIL_ImagePlaneHardwareBufferDesc struct

This struct describes a plane HardwareBuffer

typedef struct DIL_ImagePlaneHardwareBufferDesc
{
    uint32_t width;                 /* width, which might have extra padding, NOTE: the active width is in the parent Image struct */
    uint32_t height;                /* height, which might have extra padding, NOTE: the active height is in the parent Image struct */
		void*    nativeObject;          /* platform native object/id for the HardwareBuffer */
} DIL_ImagePlaneHardwareBufferDesc;

DIL Properties API

DIL_QueryProperty

Obtain a specific named property. This function provides a way of looking up a property by name to observe it's current value, this may be of use for a user to query DIL properties

DIL_ReturnCode DIL_QueryProperty(DIL_Decoder decoder, const char* name, DILProperty* output);

DIL_SetProperty

Set a specific named property. This function provides a way of changing a property by name

DIL_ReturnCode DIL_SetProperty(DIL_Decoder decoder, const char* name, DILProperty* value);

DIL_QueryPropertyGroup

Obtain all the available properties for a named group

DIL_ReturnCode DIL_QueryPropertyGroup(DIL_Decoder decoder, const char* name, DILPropertyGroup* output);

DIL_ReleaseProperty

Release a previously queried property, this function provides a way of releasing any memory allocated by a call to DIL_QueryProperty. Any data contained in value maybe invalidated after this call

DIL_ReturnCode DIL_ReleaseProperty(DIL_Decoder decoder, DILProperty* value);

DIL_ReleasePropertyGroup

Release a previously queried property group, this function provides a way of releasing any memory allocated by a call to DIL_QueryPropertyGroup Any data contained in value maybe invalidated after this call

DIL_ReturnCode DIL_ReleasePropertyGroup(DIL_Decoder decoder, DILPropertyGroup* value);

DILProperty struct.

This structure contains all relevant information used to query a property or metadata from the DIL.

typedef struct DILProperty
{
  const char*     name;        /* The name of the property.  */
  const char*     description; /* A description of the property. */
  DILPropertyType type;        /* The value type for this property. */
  union
  {
    int8_t      i8;
    int16_t     i16;
    int32_t     i32;
    int64_t     i64;
    uint8_t     u8;
    uint16_t    u16;
    uint32_t    u32;
    uint64_t    u64;
    float       f;
    double      d;
    const char* str;
    const void* vptr;
    const void* data;
		const DILPropertyGroup* pgptr;
  } value;                  /* A union representing the value of the property. The user must read the field that match the type enum value. */

  const uint8_t* blob;      /* Special field containing a block of contiguous memory when the type enum is set to DIL_PT_Blob. This is allowed to be NULL. */
  uint32_t       blob_size; /* When the blob is not NULL then the length in bytes of the memory, otherwise 0. */
	uint64_t       data_size; /* size of the "data" buffer in bytes */
	bool           can_free;  /* flag to show if the pointer values should be free'ed when the DIL_ReleaseProperty is called */
} DILProperty;

DILPropertyGroup struct.

This structure contains a logical grouping of properties, this is a convenience feature to assist with providing a "pretty" command line, or GUI.

All properties are assigned to a group, and properties can only appear in one group.

typedef struct DILPropertyGroup_t
{
  const char*  name;           /* The name for this group of properties. */
  const char*  description;    /* A description of the property group. */
  DILProperty* properties;     /* An array of properties for this group. */
  uint32_t     property_count; /* The length of the properties array. */
	bool         can_free;       /* flag to show if the properties pointer should be free'ed when the DIL_ReleaseProperty is called */
} DILPropertyGroup;