How to use =========== .. toctree:: :maxdepth: 3 :caption: Contents: Platform support ---------------- Framework: - Arduino This library is made to be used with: - ESP32 family - Teensy 4.X family Protobuf logger ---------------- The logging systems uses protobuf serialization to log data in a binary file. The binary file is composed timestamped LoggerFrames defined in the protobuf files of libDM_protobuf. The documentation of the protobuf messages can be found here: - `libDM_protobuf <../libdm_protobuf/index.html>`_ - `Protobuf definitions <../libdm_protobuf/_static/protofiles/protobuf.html>`_ The logger frame is currently defined in frame.proto file as: .. code-block:: protobuf :linenos: message LoggerFrame { // Start byte of the data frame. uint32 start_byte = 1; // Timestamp of the logger frame in us. uint64 timestamp = 2; // Header of the logger frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding optional bytes binaryHeader = 3 [(nanopb).max_size = 4096]; // Header of the logger frame (with variables types), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding optional bytes binaryTypes = 4 [(nanopb).max_size = 4096]; // Binary data of the logger frame. optional bytes binaryData = 5 [(nanopb).max_size = 4096]; // Header of the protobuf frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding optional bytes protobufHeader = 6 [(nanopb).max_size = 256]; // Binary data of the logger protobuf logged messages (RX or TX) optional bytes protobufData = 7 [(nanopb).max_size = 512]; // Header of the mavlink frame (with variables names), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding optional bytes mavlinkHeader = 8 [(nanopb).max_size = 512]; // Header of the mavlink frame (with variables ID), written one time at the beginning of the log file. char encoded as bytes to avoid fixed size encoding optional bytes mavlinkID = 9 [(nanopb).max_size = 256]; // Binary data of the logger mavlink logged messages (RX or TX) optional bytes mavlinkData = 11 [(nanopb).max_size = 280]; // Binary data of the ulog messages, char encoded as bytes to avoid fixed size encoding optional bytes ulogData = 12 [(nanopb).max_size = 512]; // End byte of the data frame. uint32 end_byte = 13; } Logging process ~~~~~~~~~~~~~~~~ When the user want to log text data, mavlink_event or protobuf event, he will need to call :cpp:func:`streamLogger::printlnUlog` or :cpp:func:`streamLogger::updateProtobufProtocolLogBuffer` and provide the data to log. The data will be timestamped and added to a LoggerFrame. This loggerFrame will be added to a ring buffer with the call of :cpp:func:`streamLogger::writeLog` and this ringBuffer will be flushed regularly to the binary file. .. image:: images/protobuf_logging_operation.png :width: 100% :align: center Extract Contents ~~~~~~~~~~~~~~~~ Tools are available inside `libDM_protobuf <../libdm_protobuf/index.html>`_ repo to extract the content of the binary file. If mavlink or protobuf messagery deifnitions are modified, the user will need to regenerate the code using `generate_protobuf.sh` or `generate_mavlink.sh` scripts and recompile tools from `libDM_protobuf <../libdm_protobuf/index.html>`_ repo using `generate_protobuf.sh` or `build_tools_windows.bat`. .. image:: images/protobuf_decode_protolog.png :width: 100% :align: center The output format provided allow for easy parsing of the data in Matlab or Python. Events or ulogs can be extracted from the json using : .. code-block:: python :linenos: file = open("output.json", "r") data = json.load(file) To extract the file, just use the script inside protobuf repo/tools: .. code-block:: bash :linenos: python decode_protobuf.py log.binDM An output folder will be created and all the files extracted inside. Binary file header ---------------------- To allow decoding of the full file, a header is added at the beginning of the binary file. It contains the following information: - Variable list separated by a comma - Variable type separated by a comma For example: | SUP_bImmobility,CAP_magTimestamp,CAP_baroTimestamp,OBS_localQuatW,OBS_localQuatX, ... ,t | uint8,uint32,uint32,float,float, ... ,float Binary file framing --------------------------- The binary file is composed of n frames of the same size, with CR/LF at the end. Each frame corresponds to one snapshot of all the added variable (with :cpp:func:`streamLogger::addVariable`) for one timestamp. When calling :cpp:func:`streamLogger::updateLogBuffer`, the internal ringBuffer is appended with the new frame. To allow easy decoding and avoid reading corrupted line, one start byte and two end bytes are added to each binary message. - start byte : 126 - end byte 1: ``\r`` - end byte 1: ``\n`` How to log data ------------------------ Include files ~~~~~~~~~~~~~~ .. code-block:: cpp :linenos: #include "libDM_timer_tool.hpp" #include "libDM_stream_logger.hpp" Initialize the logger ~~~~~~~~~~~~~~~~~~~~~~ The logger is initialized using a timertool object: .. admonition:: Example .. code-block:: cpp :linenos: timerTool myTimer; SDIO_CONF sdioConf(generalConf.sdioClock, pinConf.pinSdioClk, pinConf.pinSdioCmd, pinConf.pinSdioDat0, pinConf.pinSdioDat1, pinConf.pinSdioDat2, pinConf.pinSdioDat3, generalConf.sdioMountpoint, false); streamLogger streamObj(&myTimer, sdioConf); Binary file ~~~~~~~~~~~~ The logging is done using 3 simple steps 1. Add one time the variable at the begining of the application into log using :cpp:func:`streamLogger::addVariable` .. admonition:: Example .. code-block:: cpp :linenos: streamObj.addVariable(&OBS_localQuat[0], "OBS_localQuatW"); streamObj.addVariable(&OBS_localQuat[1], "OBS_localQuatX"); streamObj.addVariable(&OBS_localQuat[2], "OBS_localQuatY"); streamObj.addVariable(&OBS_localQuat[3], "OBS_localQuatZ"); streamObj.addVariable(&OBS_dstAltitude, "OBS_dstAltitude"); streamObj.addVariable(&OBS_spdUp, "OBS_spdUp"); 2. Update the log buffer with the frequency of your choice :cpp:func:`streamLogger::updateLogBuffer`. Each call to this function will add a new frame to the log buffer. .. tip:: This function is generally called by the main (and fastest) loop of the application. .. admonition:: Example .. code-block:: cpp :linenos: streamObj.updateLogBuffer(); // The fastest and most prioritary task updates buffers .. note:: On multi-core CPU, a possibility is to use a less prioritary core to write the log to avoid flushing SD delays to impact the main loop. 3. (optional) Use a task on another thread/core to write/flush the log. :cpp:func:`streamLogger::writeLog` the argument false for writing log. The method :cpp:func:`streamLogger::setWriteLogTask` will allow to specofy a task handle and a size threshold, above which the task will be called to write the log. .. admonition:: Example .. code-block:: cpp :linenos: void taskWriteLog(void* pvParameters) // NOLINT(misc-unused-parameters) { streamObj.setWriteLogTask(TaskHandleWriteLog); // Write logs every ko of data while (!mainTaskStarted) delay(1); streamObj.printlnUlog(true, true, "Write log task: Started"); while (true) { ulTaskNotifyTake(pdTRUE, portMAX_DELAY); streamObj.writeLog(); } vTaskDelete(NULL); } Extracting the data ------------------------ All the tools are now regrouped inside the `libDM_protobuf <../libdm_protobuf/index.html>`_ repo. A call to decode_protobuf.py will extract the data from the binary file and create all json files, csv/mat files. The binary file can be decoded using this compiled code: - Linux application :download:`Linux <_static/attached/parse_log_linux>` - MacOs apple Silicon application :download:`Apple <_static/attached/parse_log_macos>` - Windows application :download:`Windows <_static/attached/parse_log.exe>` An example file can be used to test the script: :download:`log1.binDM <_static/attached/log1.binDM>` The application is used as follows: - Mat file: .. code-block:: bash :linenos: ./parse_log -input_file log1.binDM -output_format mat -output_file output.mat - Csv file: .. code-block:: bash :linenos: ./parse_log -input_file log1.binDM -output_format csv -output_file output.csv .. note:: The mat file creation is around 10x faster than the csv file creation.