ddl_utility  1.4.3
Header2DDL

Short description: This tool chain transforms C/C++ structs and enumerations into a DDL description file

Long description: This tool chain compiles a set of C/C++ source files using a tool called CastXML which outputs its C/C++ Abstract Syntax Tree into an intermediate XML file. Most importantly this XML file comprises essential information such as names, types, alignments and sizes required for a representation of the native structs memory layout. The aforementioned XML file is then being transformed into a DDL description file allowing runtime interpretation of the struct data, endianess conversion and deserialization.

Tool Chain Interaction

source files > castxml executable > xml file > header2ddl executable > description

Example

Input Header

Consider a source file vector3d.h containing structs that you want to describe in DDL:

namespace a {
namespace b {
namespace c {
struct tVector3D {
double x;
double y;
double z;
};
} // namespace c
} // namespace b
} // namespace a

Output Description

The final description file will look similar to the following:

<?xml version="1.0"?>
<ddl:ddl xmlns:ddl="ddl">
...
<structs>
<struct name="a::b::c::tVector3D" version="1" alignment="8">
<element name="x" type="tFloat64" arraysize="1">
<serialized bytepos="0" byteorder="LE" />
<deserialized alignment="8" />
</element>
<element name="y" type="tFloat64" arraysize="1">
<serialized bytepos="8" byteorder="LE" />
<deserialized alignment="8" />
</element>
<element name="z" type="tFloat64" arraysize="1">
<serialized bytepos="16" byteorder="LE" />
<deserialized alignment="8" />
</element>
</struct>
</structs>
...
<ddl:ddl xmlns:ddl="ddl">

Intermediate CastXML Representation

The intermediate representation holds even more information:

<?xml version="1.0"?>
<CastXML format="1.1.6">
...
<Struct id="_1" name="Vector3D" context="_2" location="f1:3" file="f1" line="3" members="_3 _4 _5 _6 _7 _8 _9" size="192" align="64"/>
<Field id="_3" name="x" type="_10" context="_1" access="public" location="f1:5" file="f1" line="5" offset="0"/>
<Field id="_4" name="y" type="_10" context="_1" access="public" location="f1:6" file="f1" line="6" offset="64"/>
<Field id="_5" name="z" type="_10" context="_1" access="public" location="f1:7" file="f1" line="7" offset="128"/>
...
<Namespace id="_2" name="::"/>
...
<File id="f1" name="vector3d.h"/>
</CastXML>

Explanation

  • Struct This is the struct's element comprising its name and file id
  • Field These are the struct's field elements with among others its name, referencing the struct via the context attribute
  • File This is the source file of the compilation referenced by via the struct element's file attribute

Supported C/C++ Language Features

✔️ C/C++ standard fundamental types

✔️ Transitive struct and enum translation into description file

✔️ Type selection and therefore output description reduction

✔️ Fully qualified namespaces preserved

✔️ Anonymous structs/enums supported

✔️ Typedefs supported

Supported DDL Features

✔️ Dynamic arrays

✔️ Endianess definition

Unsupported C/C++ Language Features

Inheritance

Templates

Classes

private or protected fields

Warnings

⚠️ Fundmental types are supported but keep in mind that some are compiler/platform specific

⚠️ we highly recommend using the cross-platform compatible types from "cstdint" e.g. uint8_t, uint16_t, uint32_t, uint64_t, int8_t, int16_t, int32_t, int64_t

⚠️ bit-fields are tolerated but currently substituted with their underlying fundamental data type

⚠️bit-fields are implementation-defined or in other words compiler/platform specific!

⚠️ unions are tolerated but unsupported in DDL and are therefore substituted with a byte array

⚠️unions are implementation-defined or in other words compiler/platform specific!

⚠️unions require manual decoding measures

⚠️ If you manipulate your native compilation you must make the same manipulation in the CastXML compilation step

⚠️ compiler/platform specific descriptions/data must be decoded using DDL decoder classes

CMake Functions

header2description

This convenience function simply forwards all its argument to the following two CMake functions, therefore its signature is the combination of signatures described below.

function(header2description)
...
header2xml(${ARGN})
xml2description(${ARGN})
endfunction()

For a proper behavior it is important to use exactly the keywords described in the tables of header2xml and xml2description. While it is not necessary to keep the order of keywords, it is recommended to define first all one_value_keywords and then all multi_value_keywords.

CMake Example

header2description(
#### one_value_keywords ####
XML ${FILENAME_XML}
DESCRIPTION ${FILENAME_DESCRIPTION}
BYTEORDER "BE"
#### multi_value_keywords ####
SOURCES ${FILENAMES_SOURCES}
INCLUDE_DIRS
${INCLUDE_DIRS}
TYPES
a::b::c::tExampleA
DYNAMIC_ARRAYS
a::b::c::tExampleA::array_1[array_1_size]
HEADER2XML_ARGS
${CUSTOM_COMPILER_FLAGS})

Since the xml-file is an intermediate result, which might unimportant to the user, the specification of XML ${FILENAME_XML} can be omited. In this case, the name for the xml-file will be derived from the specification of the description file (DESCRIPTION ${FILENAME_DESCRIPTION})

CastXML CMake Functions

header2xml

This functions wraps the CastXML's executable's command line interface and therefor creates a custom command with dependencies to the input source files. It internally uses CMake's cmake_parse_arguments function to parse its arguments, therefore see the following signature:

cmake_parse_arguments(castxml "" "XML" "SOURCES;INCLUDE_DIRS;TYPES;HEADER2XML_ARGS" ${ARGN})
Keyword Required CMake Keyword Type Description
XML yes one_value_keyword The output file containing the compiled source's AST as XML
SOURCES yes multi_value_keyword The input source files
INCLUDE_DIRS no multi_value_keyword The include directories required for the input sources
TYPES no multi_value_keyword Specific types which shall be prioritized
HEADER2XML_ARGS no multi_value_keyword Additional compiler definitions | i.e. -D WIN32

⚠️ Keep in mind that the CastXML executable is a clang cross compiler.

Header2DDL CMake Functions

xml2description

This functions wraps the header2ddl executable's command line interface and therefor creates a custom command with dependencies to the input xml file. It internally uses CMake's cmake_parse_arguments function to parse its arguments, therefore see the following signature:

cmake_parse_arguments(header2ddl "" "XML;DESCRIPTION;BYTEORDER;TESTSUITE" "SOURCES;TYPES;DYNAMIC_ARRAYS;HEADER2DDL_ARGS" ${ARGN})
Keyword Required CMake Keyword Type CLI
XML yes one_value_keyword -x | –xml
DESCRIPTION yes one_value_keyword -d | –description
BYTEORDER no one_value_keyword -b | –byteorder (defaults to system endianess)
TESTSUITE no one_value_keyword -ts | –testsuite
SOURCES yes multi_value_keyword -s | –sources
TYPES no multi_value_keyword -t | –types
DYNAMIC_ARRAYS no multi_value_keyword -a | –arrays
HEADER2DDL_ARGS no multi_value_keyword all further CLI commands possible | i.e. –force-datatypes ddltypes

Example CMake Structure

It is suggested creating the generated files within the build folder (e.g. CMAKE_CURRENT_BINARY_DIR or CMAKE_BINARY_DIR) and adding them to your target's sources afterwards

# get the source file/s absolute filename
get_filename_component(FILENAME_HEADER ${CMAKE_CURRENT_LIST_DIR}/vector3d.h ABSOLUTE)
# get the source file/s name without extension
get_filename_component(FILENAME_HEADER_WE ${FILENAME_HEADER} NAME_WE)
# create intermediate XML filename based on current context and input header name
get_filename_component(FILENAME_XML ${CMAKE_CURRENT_BINARY_DIR}/${FILENAME_HEADER_WE}.xml ABSOLUTE)
# create output description filename based on current context and input header name
get_filename_component(FILENAME_DESCRIPTION ${CMAKE_CURRENT_BINARY_DIR}/${FILENAME_HEADER_WE}.description ABSOLUTE)
# compiler the sources
header2xml(XML ${FILENAME_XML} SOURCES ${FILENAME_HEADER})
# transform XML into description file
xml2description(XML ${FILENAME_XML} DESCRIPTION ${FILENAME_DESCRIPTION} SOURCES ${FILENAME_HEADER})
# target name
set(TARGET_NAME example_target)
# ...
# your target's CMake code
# ...
# add the generated sources to the target's source list
target_sources(${TARGET_NAME} PRIVATE ${FILENAME_XML} ${FILENAME_DESCRIPTION})
# display generated file in a "folder" named "generated"
source_group("generated" FILES ${FILENAME_XML} ${FILENAME_DESCRIPTION})

Command Line Interface

Although the header2ddl tool is intended to be used within a CMake context it can be treated as a standalone command line application The header2ddl command line applicatoin offers 3 modes

  • from_source: use this mode to transform header file(s) to a description in one step. This is comparable to header2description in CMake context. To use this mode, you first have to configure the tool (see config mode)
  • from_xml: use this mode to transform the intermediate XML output of CastXML to a description file. ⚠️This in only recommended, when the intermediate XML file already exists. Ohterwise you have to create it by yourself. It is highly recommended having a look into the header2xml CMake function's implementation on how to construct a valid CastXML command line call.
  • config: Specify the pathes to castxml and compiler executables on the system. These pathes are necessary to execute step one of the conversion.

In order to view the header2ddl command line argument execute the following command:

Usage:
SYNOPSIS
config -c <castxml> -p <prompt>
from_source -d <description> -s <sources>... [-t <types>...] [-a <arrays>...] [-b (LE|BE)]
[-fd (stdtypes|ddltypes)] [-i <include-dirs>...] [-ca <castxml-args>...]
from_xml -x <xml> -d <description> -s <sources>... [-t <types>...] [-a <arrays>...] [-b
(LE|BE)] [-fd (stdtypes|ddltypes)]
-h
OPTIONS
-c, --castxml
path to castXML executable 'castxml.exe'
-p, --prompt
path to MSVC command prompt 'VsDevCmd.bat' (usually under <Program
Files>/Microsoft Visual Studio/../Common7/Tools/)
-d, --description
output *.description file
-s, --sources
filtering sources to optimize the parsing speed and limit the output
-t, --types filtering types names to optimize the parsing speed and limit the output
(namespaces must be included)
-a, --arrays
list of dynamic arrays with their size field in brackets e.g.
'<namespace>::<namespace>::<struct>::<array>[<size>]' (complete namespace must
be used as prefix)
-b, --byteorder
defines the endianess used in the resulting DDL (defaults to LE)
-fd, --force-datatypes
forces the usage of 'stdtypes' (bool, uint8_t, int8_t ...) or 'ddltypes'(tBool,
tUInt8, tInt8 ...) within the description file (default is 'use as is')
-i, --include-dirs
include directories required for the input sources
-ca, --castxml-args
Additional compiler definitions for calling castxml
-x, --xml input *.xml file to be parsed
-d, --description
output *.description file
-s, --sources
filtering sources to optimize the parsing speed and limit the output
-t, --types filtering types names to optimize the parsing speed and limit the output
(namespaces must be included)
-a, --arrays
list of dynamic arrays with their size field in brackets e.g.
'<namespace>::<namespace>::<struct>::<array>[<size>]' (complete namespace must
be used as prefix)
-b, --byteorder
defines the endianess used in the resulting DDL (defaults to LE)
-fd, --force-datatypes
forces the usage of 'stdtypes' (bool, uint8_t, int8_t ...) or 'ddltypes'(tBool,
tUInt8, tInt8 ...) within the description file (default is 'use as is')
-h, --help show the usage if this tool and exit

Further Reading

Upcoming Features

In the future this tool chain will provide a code synthesis for compile time reflection code

Example Synthesized Reflection Code

template <>
const structure<tExample>& get_structure<a::b::c::Vector3D>()
{
static const auto result = structure<a::b::c::Vector3D>("a::b::c::Vector3D")
.Add("x", &a::b::c::Vector3D::x)
.Add("y", &a::b::c::Vector3D::y)
.Add("z", &a::b::c::Vector3D::z)
return result;
}

Copyright © VW Group. All rights reserved.
Generated on Thu Oct 10 2024 by doxygen 1.9.1
GIT Commit Hash: 3bf69cb9cc4cc5a5d2b26a142a474f0d9acc9ac7