How to use

Platform support

Framework:

  • Arduino

This library is made to be used with:

  • ESP32 family

  • Teensy 4.X family

Protobuf logger

The logging systems uses protobuf serialization to log data in a binary file. The binary file is composed timestamped LoggerFrames defined in the protobuf files of libDM_protobuf. The documentation of the protobuf messages can be found here:

The logger frame is currently defined in frame.proto file as:

 1message LoggerFrame
 2{
 3    // Start byte of the data frame.
 4    uint32 start_byte = 1;
 5
 6    // Timestamp of the logger frame in us.
 7    uint64 timestamp = 2;
 8
 9    // Header of the logger frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding
10    optional bytes binaryHeader = 3 [(nanopb).max_size = 4096];
11
12    // Header of the logger frame (with variables types), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding
13    optional bytes binaryTypes = 4 [(nanopb).max_size = 4096];
14
15    // Binary data of the logger frame.
16    optional bytes binaryData = 5 [(nanopb).max_size = 4096];
17
18    // Header of the protobuf frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding
19    optional bytes protobufHeader = 6 [(nanopb).max_size = 256];
20
21    // Binary data of the logger protobuf logged messages (RX or TX)
22    optional bytes protobufData = 7 [(nanopb).max_size = 512];
23
24    // Header of the mavlink frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding
25    optional bytes mavlinkHeader = 8 [(nanopb).max_size = 512];
26
27    // Header of the mavlink frame (with variables ID), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding
28    optional bytes mavlinkID = 9 [(nanopb).max_size = 256];
29
30    // Binary data of the logger mavlink logged messages (RX or TX)
31    optional bytes mavlinkData = 11 [(nanopb).max_size = 280];
32
33    // Binary data of the ulog messages, char encoded as bytes to avoid fixed size encoding
34    optional bytes ulogData = 12 [(nanopb).max_size = 512];
35
36    // End byte of the data frame.
37    uint32 end_byte = 13;
38}

Logging process

When the user want to log text data, mavlink_event or protobuf event, he will need to call streamLogger::printlnUlog() or streamLogger::updateProtobufProtocolLogBuffer() and provide the data to log. The data will be timestamped and added to a LoggerFrame. This loggerFrame will be added to a ring buffer with the call of streamLogger::writeLog() and this ringBuffer will be flushed regularly to the binary file.

_images/protobuf_logging_operation.png

Extract Contents

Tools are available inside libDM_protobuf repo to extract the content of the binary file. If mavlink or protobuf messagery deifnitions are modified, the user will need to regenerate the code using generate_protobuf.sh or generate_mavlink.sh scripts and recompile tools from libDM_protobuf repo using generate_protobuf.sh or build_tools_windows.bat.

_images/protobuf_decode_protolog.png

The output format provided allow for easy parsing of the data in Matlab or Python. Events or ulogs can be extracted from the json using :

1file = open("output.json", "r")
2data = json.load(file)

To extract the file, just use the script inside protobuf repo/tools:

1python decode_protobuf.py log.binDM

An output folder will be created and all the files extracted inside.

Binary file header

To allow decoding of the full file, a header is added at the beginning of the binary file. It contains the following information: - Variable list separated by a comma - Variable type separated by a comma

For example:

SUP_bImmobility,CAP_magTimestamp,CAP_baroTimestamp,OBS_localQuatW,OBS_localQuatX, … ,t
uint8,uint32,uint32,float,float, … ,float

Binary file framing

The binary file is composed of n frames of the same size, with CR/LF at the end. Each frame corresponds to one snapshot of all the added variable (with streamLogger::addVariable()) for one timestamp. When calling streamLogger::updateLogBuffer(), the internal ringBuffer is appended with the new frame.

To allow easy decoding and avoid reading corrupted line, one start byte and two end bytes are added to each binary message.

  • start byte : 126

  • end byte 1: \r

  • end byte 1: \n

How to log data

Include files

1#include "libDM_timer_tool.hpp"
2#include "libDM_stream_logger.hpp"

Initialize the logger

The logger is initialized using a timertool object:

Example

 1timerTool myTimer;
 2SDIO_CONF sdioConf(generalConf.sdioClock,
 3                pinConf.pinSdioClk,
 4                pinConf.pinSdioCmd,
 5                pinConf.pinSdioDat0,
 6                pinConf.pinSdioDat1,
 7                pinConf.pinSdioDat2,
 8                pinConf.pinSdioDat3,
 9                generalConf.sdioMountpoint,
10                false);
11streamLogger streamObj(&myTimer, sdioConf);

Binary file

The logging is done using 3 simple steps

  1. Add one time the variable at the begining of the application into log using streamLogger::addVariable()

Example

1streamObj.addVariable(&OBS_localQuat[0], "OBS_localQuatW");
2streamObj.addVariable(&OBS_localQuat[1], "OBS_localQuatX");
3streamObj.addVariable(&OBS_localQuat[2], "OBS_localQuatY");
4streamObj.addVariable(&OBS_localQuat[3], "OBS_localQuatZ");
5
6streamObj.addVariable(&OBS_dstAltitude, "OBS_dstAltitude");
7streamObj.addVariable(&OBS_spdUp, "OBS_spdUp");

2. Update the log buffer with the frequency of your choice streamLogger::updateLogBuffer(). Each call to this function will add a new frame to the log buffer.

Tip

This function is generally called by the main (and fastest) loop of the application.

Example

1streamObj.updateLogBuffer(); // The fastest and most prioritary task updates buffers

Note

On multi-core CPU, a possibility is to use a less prioritary core to write the log to avoid flushing SD delays to impact the main loop.

  1. (optional) Use a task on another thread/core to write/flush the log. streamLogger::writeLog() the argument false for writing log.

The method streamLogger::setWriteLogTask() will allow to specofy a task handle and a size threshold, above which the task will be called to write the log.

Example

 1void taskWriteLog(void* pvParameters) // NOLINT(misc-unused-parameters)
 2{
 3    streamObj.setWriteLogTask(TaskHandleWriteLog); // Write logs every ko of data
 4    while (!mainTaskStarted)
 5        delay(1);
 6    streamObj.printlnUlog(true, true, "Write log task: Started");
 7
 8    while (true)
 9    {
10        ulTaskNotifyTake(pdTRUE, portMAX_DELAY);
11        streamObj.writeLog();
12    }
13    vTaskDelete(NULL);
14}

Extracting the data

All the tools are now regrouped inside the libDM_protobuf repo. A call to decode_protobuf.py will extract the data from the binary file and create all json files, csv/mat files.

The binary file can be decoded using this compiled code:

  • Linux application Linux

  • MacOs apple Silicon application Apple

  • Windows application Windows

An example file can be used to test the script: log1.binDM

The application is used as follows:

  • Mat file:

1./parse_log -input_file log1.binDM -output_format mat -output_file output.mat

  • Csv file:

1./parse_log -input_file log1.binDM -output_format csv -output_file output.csv

Note

The mat file creation is around 10x faster than the csv file creation.