How to use

Introduction

This repo contains all the log processing tools centralized. This tools can use user generated protobuf mavlink content. They are composed of cpp buildable files and python scripts. Shell and bat scripts are also provided to build the cpp files for windows or unix platforms.

• The logging mechanism is described in the stream logger library: libDM_stream_logger.

• The protobuf mechanism used is described in the protobuf library: libDM_protobuf.

The proposed logging mechanism allows unique file logging for: - protobuf events - mavlink events - binary logs - text logs …

parse_binDM

This part of the repo contains the tools to parse the binary logs logged inside the protolog. As this kind of logs are often high frequency with a lots of variables, the tools are optimized to be fast and efficient. The main script used is located in /parse_binDM/src/parse_log.cpp

It can be recompiled with the provided build script located at /parse_binDM/build_unix.sh or /parse_binDM/build_windows.bat

The executable files are located in /parse_binDM/builds/xxx

The script can generates outputs in csv`or `.mat format. From a binary file.

It can be used with the following command line:

./parse_log_[platform] -i [input_file] -o [output_file] -f [format]

with :

Hint

• input_file: input file path

• output_file: output file path

• output_format: mat or csv

• debug: enable debug printf

parse_protolog

A decoding script is provided to decode completely the protobuf logs, including, events, binary logs, mavlink messages, etc. The script is located in the folder parse_protolog/tools/decode_protobuf.py.

The script can be used as follow:

• Install the requirements:

pip install -r requirements.txt

• Run the script:

python decode_protobuf_log.py --input_file log.binDM

Full options with python3 decode_protobuf_log.py -h:

usage: python3 decode_protobuf_log.py --input_file ../test/log_test.binDM --format csv --do_zip True --parse_protolog_custom_binary decode_protobufDM_linux --parse_binary_custom_binary parse_log_linux

Extract protobuf encoded logs

options:
-h, --help            show this help message and exit
--input_file INPUT_FILE
                        Input file path
--format FORMAT       Output format for binary data (mat or csv)
--do_zip DO_ZIP       Zip the extracted file?
--parse_protolog_custom_binary PARSE_PROTOLOG_CUSTOM_BINARY
                        Custom compiled protolog extractor
--parse_binary_custom_binary PARSE_BINARY_CUSTOM_BINARY
                        Custom compiled binary log extractor

The script will create a new folder containing the raw extracted data and decoded files with the indicator _decoded.

Note

If the protobuf/mavlink built files are not present or not up to date, they can be automatically imported with :

python3 get_messaging.py

The files will be imported either from the user workspace, directly taking the files from the protobuf/mavlink repos. If these folders are not present, the script will clone the current repos.

The extraction results should looks like this:

The extracted data will be stored in the output folder:

The content of the different files extracted should looks like:

• The binary log:

  

• The protobuf events:

  

• The mavlink messages:

  

• The text logs: