HHD file format

The HHD (HH Data format) is a custom archive file format that uses the Zstd compression algorithm for efficient data storage.

Archive file structure

Header

	Offset (bytes)	Data Type	Size	Description
Identifier	0	uint	4	File identifier: 0x48484431
Version	4	uint	4	File version: 1
Directory offset	8	long	8	Location of the archive directory within the file
Entry count	16	long	8	Number of archived entries
Entry data BLOBs	24	Byte Arrays	Variable	Zstd-compressed data entries
Archive directory	Directory Offset	`ArchiveEntry'	Variable	Uncompressed binary data

ArchiveEntry object type

Each archive entry contains the the information for accessing compressed data:

Field	Type	Description
Name Length	int	Length of the Name string
Name	string	Unique identifier for the archive entry of the specified length
Data offset	long	Byte offset to the compressed data
Compressed length	long	Size of compressed data in bytes
Decompressed length	long	Original size after decompression

Types of archive entries

All entries are compressed with Zstd. The following entries are present in every HHD file:

Entry Name	Content Type	Description
[FileProperties]	Serialized JSON	Serialized HHDFileProperties object
[Laps]	Serialized JSON	List of lap elapsed times (doubles)
[Constants]	Serialized JSON	List of IConstantProperties objects
[DataChunk||Channel Name]	Serialized Binary	Numeric data array for the specified channel
[ChannelProperties]	Serialized JSON	List of HHDChannelProperties objects

[ChannelProperties] JSON Schema

Field	Type	Description
Name	string	Channel identifier
Nullable	bool	Whether channel values can be null
DataType	eChannelDataType	Data type of channel values (see supported types)
ScalingFactor	double	Multiplier applied to raw values
Offset	double	Offset applied to raw values
Frequency	int	Sampling frequency (Hz)
SampleCount	int	Total number of samples in channel
Dimension	eDimension	Physical dimension (e.g., velocity, temperature)
Unit	string	Unit of measurement
TimeChannelName	string	Reference to time channel for synchronization
FirstSampleTime	double	The time of the first data sample (if not 0)

Data Chunks

Data chunks store numeric arrays with optional null support, compressed with Zstd.

• Non-nullable channels: Raw array of values of the specified data type
• Nullable channels: Series of flag bytes followed by values
  • Flag byte = true: next byte(s) contain the value in the specified data type
  • Flag byte = false: value is null, proceed to next flag byte

Data types

eChannelDataType	.NET Type	Description
SByte	byte	unsigned 8-bit
Byte	sbyte	signed 8-bit
Int16	short	signed 16-bit integer
UInt16	ushort	unsigned 16-bit integer
Int32	int	signed 32-bit integer
UInt32	uint	unsigned 32-bit integer
Single	float	32-bit IEEE 754
Double	double	64-bit IEEE 754

Example: Reading a Data Chunk

1. Open the HHD archive file and read the header to locate the archive directory offset
2. Load the directory by reading Entry count of ArchiveEntry objects
3. Select the [ChannelProperties] entry:
   • Seek to the entry's data offset in the archive
   • Read the compressed data array (size: compressed length)
   • Decompress using Zstd decompression algorithm
   • Deserialize the decompressed JSON data into a collection of HHDChannelProperties objects
4. Find the specific HHDChannelProperties object matching your target channel name
5. Use the channel's metadata to process the corresponding data chunk:
   • Decompress the [DataChunk||Channel Name] entry using Zstd
   • Parse the byte array based on the channel's Nullable flag and DataType:
     • If nullable: read flag byte, then value byte(s) if flag is true
     • If non-nullable: read contiguous values in the specified DataType
6. Apply ScalingFactor and Offset transformations if needed

Scaling Factor and Offset

Channel data in HHD is represented on disk using numeric primitives (integers or floats) while application code typically treats channel values as nullable double. The ScalingFactor and Offset fields convert between the stored representation and the application double values.

• Conversions
  • From stored value to double (stored -> double):
    • doubleValue = (storedValue - Offset) / ScalingFactor
    • Apply this only to non-null values. If a value is marked null in a nullable channel, keep it as null.
  • From double to stored value (double -> stored):
    • storedValue = (doubleValue * ScalingFactor) + Offset

Example: writing an HHDFile without the HHDFileWriter or ZstdDataRepository

1. Write the file header with placeholders for Directory offset and Entry count
2. Write file-level properties (entry name: [FileProperties]):
   • Serialize a HHDFileProperties object JSON
   • Compress and store in the archive
     • Write the compressed byte array to the file
     • Cache an ArchiveEntry object in directory collection to write later when finalizing the archive.
3. Write lap elapsed times ([Laps]):
   • Serialize a list of doubles into a JSON string
   • Compress and store in the archive
4. Write constant properties ([Constants]):
   • Serialize a list of IConstantProperties objects into a JSON string
   • Compress and store in archive
5. For each channel, write channel data ([DataChunk||CHANNEL NAME]):
   • It is optional to convert channel values from double?[] to a smaller storage type using ScalingFactor and Offset
   • Serialize the value arrays to byte arrays:
     • Nullable arrays consist of a boolean flag to indicate if the following bytes contain this element's value or the next element of the array (e.g. one array element can be a single-byte boolean false, followed by the next element's boolean true and an 8-byte double value)
     • Non-nullable arrays consist of contiguous stored-type values (e.g. Double is 8 bytes, UInt16 is 2 bytes)
   • Compress the byte array with Zstd and add to archive
6. Write channel properties index ([ChannelProperties]):
   • Create a list of HHDChannelProperties metadata objects (name, data type, frequency, units, scaling)
   • Serialize the list to a JSON string
   • Compress and store in archive
7. Finalize the archive:
   • Write the archive directory (reference the ArchiveEntry type definition and do not compress)
   • Updated the header with Directory offset and Entry count values
   • Close the file