as comfortable and extendable command line tooling.
usage:
adtf_dattool options
where options are:
-?, -h, --help display usage information
--progress Show progress.
--skipstreamtypesandtriggers Do not process stream types and
triggers.
--plugin <adtffileplugin> Load an additional adtffileplugin,
e.g. to support fiel formats
different from adtfdat.
--inspect <adtffileplugin> Inspect a adtffileplugin and print
its contained items.
--inspect-builtins Inspect all built-in classes.
--log-level <loglevel> Set the log level
--json Output information in json format.
Raw timestamps within the json
output are always output as seconds.
Redirects log messages to stderr.
--liststreams <file name> List all available streams about the
given adtfdat file.
--listsubstreams <stream name> List all substreams available in the
given stream.
--export <file name> Export streams from the given
adtfdat file.
--create <file name> Create a new adtfdat file.
--modify <file name> Modify an existing adtfdat file.
--dump <file name> Dump information about an adtfdat
file. If no other options (--header,
--streams, --stream, --extensions, -
-extension) are additionally
specified, an entire overview will
created.
--header Show header information for a
adtfdat file in combination with --
dump.
--streams Show all stream information for a
adtfdat file in combination with --
dump.
--extensions Show all extension information for a
adtfdat file in combination with --
dump.
--chunks Show all chunk information for a
adtfdat file in combination with --
dump.
--statistics <file name> Calculate statistics on samples,
trigger and stream types. Use --
stream, --substream and --entity to
limit output to a specific set.
--entity <entity> Report only the given statistics
entity. Possible arguments are
samples, triggers, stream_types,
samples_per_sec, triggers_per_sec
and stream_types_per_sec.
--fileversion <adtf2|adtf3|adtf3ns> File Version of the created file.
Creating ADTF 2 files is completely
experimental!
--store-checksums When creating a new adtfdat file,
calculate checksums and store them
in the 'checksums' extension.
--input <source> Specifies an input for the new
adtfdat file or extension. In case
of extensions data will be read from
stdin if this is not specified.
--readerid <reader id> The id of the reader implementation
that should be used to open the last
list/input/export source.
--referencedfiles Process referenced files (i.e.
splits) from the last input/export
source as well.
--ignoreopenerrors Ignore errors when referenced files
cannot be opened.
--start <timestamp> Process only chunks of the last
input source with timestamps larger
than this. @See List of supported
timestamp formats.
--end <timestamp> Process only chunks of the last
input source with timestamps smaller
than this. @See List of supported
timestamp formats.
--offset <timestamp> This offset is added to all chunk
timestamps of the last input source.
@See List of supported timestamp
formats.
--stream <stream name> Select a stream for dump, export,
modify existing or create new
adtfdat file.
--stream-type <stream name> Dump stream type of the given
stream.
--ddl <stream name> Dump ddl from stream type of the
given stream.
--substream <substream name> Select a substream for export. Keep
in mind that if you want to export
multiple substreams from the same
stream, you need to specify --stream
for each substream. If the substream
is not available within the initial
stream type you always need to
specify a processor id with --
processorid.
--name <stream name> Sets the name for the last specified
stream that is used in the newly
created adtfdat file.
--processorid <processor id> The id of the processor
implementation used for the last
specified stream or substream.
--property <name=value> Sets a property of the reader for
the last specified export job, dump
job, statistics job or create job
input or export processor of a
stream.
--extension <extension name> Select an extension for dump,
export, modify existing or create a
new adtfdat file.
--output <file name> Sets the output file name for the
last specified stream or extension.
If not specified, the processor is
free to choose one and in case of
extensions data will be written to
stdout
--serializerid <serializer id> The id of the serializer
implementation used for the last
specified stream.
--userid <user id> Sets the user id of the last
extension to be updated.
--typeid <type id> Sets the type id of the last
extension to be updated.
--versionid <version id> Sets the version id of the last
extension to be updated.
--rename <new name> Renames the given stream.
--repair <file name> Repair the given corrupted file.
Requires --reference and --output as
well.
--reference <file name> The reference file for the repair
operation.
--verify-integrity <file name> Verify the integrity of the given
file by recalculating and comparing
the checksums contained in the
'checksums' extension.
-----------------
ADTF DAT Tool
-----------------
With the help of this tool you can extract data from adtfdat files, create new adtfdat files
from various inputs or add file extensions to an existing adtfdat file.
Use '--liststreams' to query all information about a given input (adtfdat file, or any other
supported input).
To load additional adtffileplugins use the '--plugin <adtffileplugin>' argument as often as you like.
-----------
EXPORT:
-----------
To extract data from streams or file extensions of an adtfdat file use the '--export <filename>' argument.
Select the streams you want to extract by using the '--stream <stream name>' argument.
You can specify a processor for each stream with the '--processorid <processor id>' argument.
If you do not specify one explicitly, the first one that supports the stream is used.
Properties of processors can be specified with the '--property <property name>' argument.
Use the '--extension <extension name>' argument to specify the required file extension.
The destination filename is specified with the '--output <filename>' argument.
Without '--output' argument the file extension data will be written to stdout.
To process a set of files in one go specify '--readerid multiple_files' right after your '--export <filename>' or '--input <filename>' argument and specify the set of files as a ';' seperated list.
To process referenced files transparently specify '--referencedfiles' right after your source adtfdat file.
Examples:
---------
Here is an example that exports two streams:
adtf_dattool --plugin csv_exporter.adtffileplugin --export test.dat --stream in1 --processorid csv --output test_in1.csv --stream in2 --output test_in2.csv
And one that exports extension data:
adtf_dattool --export test.dat --extension adtf_version --output adtf_version.txt
Process referenced files as well:
adtf_dattool --plugin csv_exporter.adtffileplugin --export first_split_001.dat --referencedfiles --stream in1 --processorid csv --output in1.csv
Export only part of input limited by global '--start <timestamp>' and or '--end <timestamp>':
adtf_dattool --plugin csv_exporter.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output cutted.csv --start 7.7s --end 10s
-------------
CREATION:
-------------
To create a new adtfdat file use the '--create' argument.
The '--input <filename>' argument is used to specify a source.
The '--readerid <reader id>' argument can be used to specify the reader that should be used to read the file.
If none is specified, the first one that supports the file is used.
Use the '--start <timestamp> and '--end <timestamp>' arguments to select the range of the input that should be imported into the adtfdat file.
Use the '--offset <timestamp>' parameter to shift the timstamps of all imported stream items.
To select streams from an input, use the '--stream <stream name>' argument.
If you do not select one or more streams neither a extension, all streams will be added.
Streams can be renamed with the '--name <new name>' argument.
Use the '--serializerid <serializer id>' argument to choose the serializer of your liking.
If not specified, 'sample_copy_serialization_ns.serialization.adtf.cid' will be used.
To select file extensions from an input, use the '--extension <extension>' argument.
If you do not select one or more file extensions explicitly, no extension will be added or updated from the source file.
Examples:
---------
Here is an example that creates a new adtfdat file from two inputs:
adtf_dattool --create new.adtfdat --input input1.dat --readerid adtfdat --stream in1 --serializerid special.serialization.adtf.cid --input input2.dat
And one that copies a stream and two extensions from a source file
adtf_dattool --create new.adtfdat --input input1.dat --stream in1 --extension attached_files --extension attached_files_configuration
------------------------
MODIFYING:
------------------------
To modify an existing adtfdat file use the '--modify <filename>' argument.
NOTE: Currently there is only support for adding and updating extensions.
Examples:
---------
An example to put a files content into an extension:
adtf_dattool --modify existing.adtfdat --extension my_extension --input input_file.txt
Mind that if you do not specify an --input for an extension its data will be read from stdin:
tar cz /myfolder | adtf_dattool --modify existing.adtfdat --extension attached_files
or use:
adtf_dattool --export source.adtfdat --extension attached_files | adtf_dattool --modify destination.adtfdat --extension attached_files
to copy a explicit file extension of an existing adtfdat file to another.
To rename a stream use:
adtf_dattool --modify existing.adtfdat --stream old_name --rename new_name
------------------------
DUMP:
------------------------
To show some information about an adtfdat file itself and its entire content you can use '--dump <file name>'.
This works also in combination with '--streams' or '--header' to print these specifiic stream or extension information.
Furthermore, you can dump also the content for an explicit '--stream <stream name>' or '--extension <extension name>'.
------------------------
STATISTICS:
------------------------
To analyze the streams, triggers and stream types of an adtfdat file you can use '--statistics <file name>'.
You can focus only to a single data stream by adding '--stream <stream name>' and '--substream <substream name>'.
Furthermore, you can also limit the return value to a specified '--entity <entity>'.
Finally, you are also able to trim the sequence you want to calculate statistics for by using '--start <timestamp>' and/or '--end <timestamp>'.
------------------------
INSPECTION:
------------------------
To inspect an adtffileplugin use the '--inspect <adtffileplugin>
' argument.
This will list all classes and their configuration properties contained within the given plugin file.
To list all built-in classes use '--inspect-builtins'.
------------------------
TIMESTAMP FORMATS:
------------------------
Supported are the following ISO-8601 formats:
e.g. 2024-09-26T23:59:59Z -> Universal time UTC-0 (GMT). (preferred)
e.g. 2024-09-26T23:59:59 -> Local time.
e.g. 2024-09-26T23:59:59.123456789Z
With timezone postfix 'Z' (default) means GMT time (UTC-0). Universal Coordinated Time.
No timezone postfix 'Z' means your local timezone.
Supported is the number+unit format:
with several unit postfixes = {h|hh|m|mm|min|s|ss|sec|ms|us|ns}
e.g. 14.125h
e.g. 8123ms
e.g. 3.141s
Unsupported formats:
e.g. 9.12 (must have unit)
e.g. T23:59:59 (no time only ISO timestamps)
e.g. 1h 30ms (no combined time fragments)
In this mode the ADTF DAT Tool makes use of processors to operate on streams inside the .adtfdat file.
Currently the ADTF File Library delivers one such processor to export the samples of a stream into a .csv file (see csv_exporter). There are serval further exporter available, such as .mdf, .parquet and .pcap/pcapng among others at for example digitalwerk.
This chapter should show you a workflow how to start from scratch to use the ADTF DAT Tool, extend the functionality using adtffileplugins and how to export containing streams from a recording. The separated steps will teach you about architecture and how to components and functionality are connected. For a better overview, we strip the paths from the call and arguments.
First of all, the command line help should always be the first address for information, no matter as getting started or for a deeper dive later:
As next step you should have a look at the recorded file. The --liststreams option will dump the content, in this case, one video and one ddl described stream:
With the knowledge about the containing streams, it is time to find a suitable processor to export some data. The --inspect option can be used to look inside an .adtffileplugin, in this case the csv_exporter with the ADTF File Library besides the ADTF DAT Tool itself.
Use cases for using properties for this processor could be adapting the decimal places:
Processing substreams is almost the same workflow, with tiny extensions. The --liststreams command will show you the content as already learned:
You will receive the same required information compared to the common sample streams use case. Now you can also extend the functionality using a processor and export csv in the same manner
Now you have a good start and also common workflow for the export use case. Of course you can use this procedure with less adaption for create and modify option as well.
