Readers

adtfdat

ADTF File Reader to deserialze and read the content of a ADTF DAT File.

Properties

Name	Default Value	Type	Description
ignore_unsupported_streams	false	bool	Property to ignore unsupported streams with no error (true, false)
read_cache_size	-1	signed	Property to set readcache size in bytes for cached block reading to optimize read operations of file content. Use ADTFDatFileReader::read_cache_default use the default the cache size (currently 8kB). Use ADTFDatFileReader::read_cache_deactivated deactivates the read cache

multiple_files

Abstract base class for a multi reader.

Properties

Name	Default Value	Type	Description
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

referenced_files

Reader implementation to read multiple files and its referenced files extension.

Properties

Name	Default Value	Type	Description
ignore_referenced_files_open_errors	false	bool	Ignore errors when referenced files cannot be opened.
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

sample_timestamp_sanitizer

Proxy reader that ensures sample timestamps are monotonically increasing without reordering samples. Only fixes non-monotonic sample timestamps. The logged file remains invalid in general. In contrast to 'sorted_by_timestamps', which reorders items, this reader adjusts timestamps, but preserves the original order. Using the wrong reader may further corrupt the data.

Properties

Name	Default Value	Type	Description
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

sorted_by_timestamps

Proxy reader that reorders items within a sliding window to restore timestamp order. Only fixes out-of-order items. The logged file remains invalid in general. In contrast to 'sample_timestamp_sanitizer', which adjusts non-monotonic sample timestamps, this reader reorders the items but preserves the original sample data. Using the wrong reader may further corrupt the data.

Properties

Name	Default Value	Type	Description
generate_trigger_for_each_sample	true	bool	Whether or not to emit a trigger for each sample item.
sliding_window_size_seconds	10	unsigned	The size of the sliding window whithin items are sorted in seconds.
sort_by_sample_timestamp	true	bool	Whether to sort by sample or chunk timestamps.
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

stream_renaming

Proxy Reader that allows renaming input streams based on user-defined regex rules.

Properties

Name	Default Value	Type	Description
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

streams_to_substreams

Proxy Reader that merges all streams into a single stream (named 'substreams') containing substreams.

Existing substreams will be prefixed with the name of the stream they originate from, i.e. 'streamX.substreamY'. Substreams for plain streams will be named equal to the stream itself.

Properties

Name	Default Value	Type	Description
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

time_extension

Proxy Reader that allows you to provide samples after the orignal reader has signal EOF.

Properties

Name	Default Value	Type	Description
stream_name	time_extension	string	Name of the extra stream where dummy stream items will be generated.
time_extension_interval_ns	0	signed	Time extension interval. If time_extension_interval_ns == 0 then time_extension_interval_ns will set to time_extension_ns.
time_extension_ns	0	signed	Time extension in nanoseconds. If time_extension_ns == 0 then no extra stream items will be generated.
worker_reader		string	The identifier of the reader implementation. If empty, the first compatible reader is automatically searched for. Place the configuration for the worker reader as a sub tree to this property.

Sample Serializers

sample_copy_serialization.serialization.adtf.cid
sample_copy_serialization_ns.serialization.adtf.cid
media_description_sample_serialization.serialization.adtf.cid
media_description_sample_serialization_ns.serialization.adtf.cid
adtf.core.media_sample.adtf2_support.serialization.adtf.cid

Sample Deserializers

sample_copy_serialization.serialization.adtf.cid
sample_copy_serialization_ns.serialization.adtf.cid
media_description_sample_serialization.serialization.adtf.cid
media_description_sample_serialization_ns.serialization.adtf.cid
adtf.core.media_sample.adtf2_support.serialization.adtf.cid
indexed_data_toolbox.indexed_data_media_sample.adtf2_support.serialization.adtf.cid

Type Serializers

adtf2/legacy
adtf/image
adtf/audio

Type Deserializers

adtf.core.media_type.adtf2_support.serialization.adtf.cid
adtf.type.video.adtf2_support.serialization.adtf.cid
adtf.type.audio.adtf2_support.serialization.adtf.cid
indexed_data_toolbox.indexed_data_description_media_type.adtf2_support.serialization.adtf.cid

Plugin Attributes

Attribute	Value
Plugin Name	builtins.adtffileplugin
License	MPL-2.0
Support Mail	suppo.nosp@m.rt@d.nosp@m.igita.nosp@m.lwer.nosp@m.k.net
Homepage URL	https://adtf.dev/

Processing multiple input files at once

Specifing the set of input files manually

To process multiple files as input during one go, use the multiple_files reader (with --readerid) and separate the input filenames with ; wherever you would otherwise specify a single filename only. The files need not contain the same streams.

adtf_dattool --plugin <exporter>.adtffileplugin --export "file1.adtfdat;file2.adtfdat" --readerid multiple_files --stream <stream_name> --output <exported_file>

If for some reason there are multiple readers capable of reading the input files you can select one explicitly by setting the worker_reader property. To set properties for the worker reader set them with a worker_reader/ prefix, i.e.:

adtf_dattool --plugin <exporter>.adtffileplugin --export "<file1>;<file2>" --readerid multiple_files --property worker_reader=<reader_id> --property worker_reader/<reader_property>=foo --stream <stream_name> --output <exported_file>

Using the referenced file information

The ADTFDAT file format supports linking split files via an included extension. You can use the referenced_files reader to automatically process the whole set of linked files at once:

adtf_dattool --plugin <exporter>.adtffileplugin --export "split_file1.adtfdat" --readerid referenced_files --stream <stream_name> --output <exported_file>

WARNING: The --referencedfiles option is an alias for --readerid referenced_files, so keep in mind that you cannot use both and that you need to specify properties for the worker reader with the worker_reader/ prefix in either case.

Example with the csv_exporter

Given two consecutive files with the same stream my_stream we can export the concatenated stream with a single export command as follows:

adtf_dattool --plugin csv_exporter.adtffileplugin --export "file_split1.adtfdat;file_split2.adtfdat" --readerid multiple_files --stream my_stream --output out.csv

Example with the csv_reader

Given two consecutive CSV input files we can import the concatenated data with the following command. See how we specify a property for the worker reader as well which is applied to all the input files:

adtf_dattool --plugin csv_reader.adtffileplugin --create my_file.adtfdat --input "file1.csv;file2.csv" --readerid multiple_files --property worker_reader=csv --property "worker_reader/separator=#"

Example with the stream_renaming_reader

Given an input file example_test_file.dat containing the following streams:

streams:
  video:
    stream_id: 1
    index_count: 149
    ...
  NESTED_STRUCT:
    stream_id: 2
    index_count: 297
    ...

You can rename these streams using the stream_renaming_reader and regular expression patterns. In this example:

we match the stream name video using the regex ^video$ (matches exactly "video"),
and rename it to VIDEO_STREAM.
Likewise, we rename NESTED_STRUCT to STRUCT_STREAM.

If a regex matches multiple streams, an error is raised.

The following ADTF DAT Tool command shows the renamed streams using –liststreams:

adtf_dattool --liststreams example_test_file.dat --readerid stream_renaming --property filter/VIDEO_STREAM="^video$" --property filter/STRUCT_STREAM="^NESTED_STRUCT$"

Output:

reader: stream_renaming
streams:
    stream: VIDEO_STREAM
    type: adtf/image
    item_count: 149
    ...
 
    stream: STRUCT_STREAM
    type: adtf2/legacy
    item_count: 297
    ...