Embedding Micropython in your Projects

MicroPython is a python implementation for microcontrollers and other low resource requirement implementation which can make it perfect for embedding without significantly increasing your delivered executable or firmware.

The project was created by Damien George at 2013 and has since grown and improved and even earned its place at OBCPs and is planned to reach space on board Euclid at 2023 as well as integrating into RODOS.

If it can work for spacecrafts, why shouldn't it work for you?

My original goal was to use MicroPython as an expression evaluation and enrichment engine for embedded data acquisition, however, I've found that the complexity of actually integrating it into a PlatformIO project was too big to miss on an opportunity to make it an easier task.

MicroPython Project Structure

After downloading the release (in my case 1.19), the archive has the following interesting folders:

py - contains MicroPython VM

ports - contains platform specific implementation and support functionality

extmod - extra modules

I was a bit naïve, so I've decided to build all files in py and windows port but there were so many errors I've realized its probably the wrong approach.

I've started digging in the makefiles since I wasn't sure what to look for, the instructions were pretty standard. but once you start looking in the makefiles, there's plenty going on there.

But there's no easy way to use that in PlatformIO as far as I know, so I went ahead and checked what was executing when and why, there's some documentation about qstr but there are also modules and version info, some of it to optimize the build and prevent a rebuild of the whole qstr header.

I wanted to avoid precompiling the headers and keeping them in git, since it will complicate version updates, build flags changes and built-in module changes, there have been similar approached with NXP port but I wanted it to be more robust and developer friendly.

I've decided to keep it short and just write a script that executes all the scripts on the appropriate files and folders and after some fiddling with the list of sources that needed to be compiled, the project compiled on both Windows and ESP32.

Building MicroPython on PlatformIO

PlatformIO uses scons to build, scons keeps a series of flags and dictionaries of the files to build and uses the compiler so actually generate the executable and firmware. Though in ESP32 case it uses cmake as well to build esp-idf.

PlatformIO also added hooks before and after building the project, allowing extensibility through scripting, in this case I've chosen to execute a script in the library so its always executing before the build.

I went ahead and made the build system a bit more flexible by allowing different sources and includes to be used for different platforms. You may want to look in build_settings.py to see how.

In short, the script determines the framework and platform and reads the library.json->build->environments, it appends all the sources and flags from common and then looks for a specific platform and appends the sources and flags from there as well.

I then read list of file selectors (SRC_FILTER) and build a list of source files so it can be used to generate the headers with generate_strings.py

• makeversionhdr - generates the mpversion.h
• makeqstrdefs pp - generates qstr.i.last from a batch of source files and headers
• makeqstrdefs split qstr - extracts a list of qstr from the last qstr.i.last, this may run a few times with a few different qstr.i.last
• makeqstrdefs split module - extracts a list of modules from the last qstr.i.last, this may run a few times with a few different qstr.i.last
• makeqstrdefs cat module - generates a collected file from the splitted modle files
• makeqstrdefs cat qstr - generates a collected file from the splitted qstr files
• makemoduledefs - generates moduledefs.h from the collected module file
• qstrdefs_preprocessed - precompiles strings into preprocessed.h
• qstrdefs_generated - generates generated.h from preprocessed.h

Once this process is done, the qstr and modules dictionaries are ready to be compiled into the library.

Generic Port

MicroPython uses a set of configuration header files and functions so it will be portable and work across multiple environments but to run it without any special hardware modules requires very little configuration.

I've created a "generic" port:

• mimimal gc
• unhandled exceptions are sent to stderr
• all filesystem operations are throwing errors
• all python's print are sent to stdout, stdin does nothing

Initialization

To initialize micropython we need a stack and heap and then initialize the stack, the stack limit, the garbage collector which also handles allocations and finally MicroPython.

Please note there are two stack limits, one for python (pystack) and one for the OS limit, which is used to limit the recursion the MicroPython engine uses - it should be less than the OS stack size, looking at MicroPython's code, its about 2k less than the OS stack.

1 2 3 4 5 6 7 8 9 10 11 12

instance->stack = (uint8_t *)malloc(instance->stack_size); instance->heap = (uint8_t *)malloc(instance->heap_size); mp_stack_ctrl_init(); mp_stack_set_limit(instance->stack_size); // Initialize heap gc_init(instance->heap, instance->heap + instance->heap_size); mp_pystack_init(instance->stack, instance->stack + instance->stack_size); // Initialize interpreter mp_init();

NLR

NLR stands for non-local return, which is how MicroPython handles exceptions in C, using a stack of jumps.

1 2 3 4 5 6 7 8 9 10 11 12

nlr_buf_t nlr; if (nlr_push(&nlr) == 0) { mp_obj_t retval = mp_call_function_n_kw(func, argc, 0, argv); nlr_pop(); return retval; } else { mp_obj_print_exception(&mp_stderr_print, MP_OBJ_FROM_PTR(nlr.ret_val)); return (mp_obj_t)nlr.ret_val; }

1-2 define a temporary nlr buffer and push the current context.

4 execute a MicroPython function

5 pop for successful execution

8 in case there's an error between lines 3-7 the IP will jump to line 9.

10 display the error

Executing Scripts and Calling Functions

While it seems very similar in python, there is a difference, executing a script might have an output, it is not captured, so there is no way to use that output unless you plan on capturing and parsing Python's print.

This is why I chose to split that functionality into two parts, the first part executes a script, creating functions / variables on the local/global scope and the second part executes a function and uses its return value.

For executing scripts, MicroPython needs to parse the string and compile it into MicroPython bytecode and execute it. So once the script is in the context, we can use the module for looking up the function name and execute it.

We'll pass the following script to the compilation code

1 2 3

import math; def transform(a,b): return (a**2/math.sin(2*math.pi/b))-a/2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

nlr_buf_t nlr; if (nlr_push(&nlr) == 0) { qstr src_name = 1 /*MP_QSTR_*/; mp_lexer_t *lex = mp_lexer_new_from_str_len(src_name, fragment, strlen(fragment), false); qstr source_name = lex->source_name; mp_parse_tree_t pt = mp_parse(lex, MP_PARSE_FILE_INPUT); mp_obj_t module_fun = mp_compile(&pt, source_name, false); mp_call_function_0(module_fun); nlr_pop(); return NULL; } else { mp_obj_print_exception(&mp_stderr_print, MP_OBJ_FROM_PTR(nlr.ret_val)); return (mp_obj_t)nlr.ret_val; }

And now we can lookup transform (line 1) function and call it (line 10)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

mp_obj_t transform_function = mp_obj_dict_get(mp_locals_get(), mp_obj_new_str("transform", strlen("transform"))); mp_obj_t args[2]; args[0] = mp_obj_new_float(1); args[1] = mp_obj_new_float(2); nlr_buf_t nlr; if (nlr_push(&nlr) == 0) { mp_obj_t retval = mp_call_function_n_kw(transform_function, argc, 0, argv); nlr_pop(); return retval; } else { mp_obj_print_exception(&mp_stderr_print, MP_OBJ_FROM_PTR(nlr.ret_val)); return (mp_obj_t)nlr.ret_val; }

Summary

MicroPython has advanced light years since its first inception and it is very capable and enables rapid prototyping and even usable for space applications. That being said, its still a bit slow to function as a generic programming language and more appropriate for coordinating calls to C functions.

The other thing I've found lacking is documentation, to write this demo I needed to go over multiple sources and demos, look for forks, pull requests and read a lot of source code, its still not 100% clear to me what all the defines are doing exactly and what to expect in terms of performance impact for each of them.

In my particular use case:

• x64 native vs MicroPython slowdown is about x16 times.
• ESP32 native vs MicroPython slowdown is about x100 times.

For the curious minds of where did the CPU spent its time, I'm very happy to have found Very Sleepy

As always you can find the fruits of my labor at my GitHub account.

Further reading:

• https://www.snaums.de/static/resources/2017-12-mpy.pdf
• https://micropython.org/resources/publish/Spacebel_MicroPythonOBCP_Dasia2018_Paper.pdf
• https://mediatum.ub.tum.de/doc/1519584/document.pdf
• https://indico.esa.int/event/269/contributions/4156/attachments/3161/4272/Presentation_-_Qualification_of_a_Virtual_Machine_an_OBCP_Engine_Based_on_MicroPython.pdf

Tags: micropython, platformio, scripting, very sleepy