Unit Testing with ESP32 and PlatformIO

Writing Quality Code and Tests usually goes hand in hand, some find it more productive to write code and then write tests for it and others, following TDD/BDD write tests before code. The benefits of each is out of scope for this article, what is in scope is how to reach the nirvana of software development by using tests.

source

Lets start with the basics.

PlatformIO tests can be run on:

• Locally - on the development machine, it requires that gcc is installed and in the path. for windows this can be done with mingw gcc.
• On Device - PlatformIO automates this by building the tests, uploading them and resetting the device in order for them to execute and wait for 2 lines, one stating how many tests were executed and another stating if any of the tests failed.
• Remotely - PlatformIO automates this by building the code on the development machine, copying the firmware to the remote agent, uploading to the device connected to the remote agent and waiting for the test results.

To reach that nirvana we can write tests that execute on the ESP32, but no matter how fast your machine is, compiling, uploading and executing tests on ESP32 can take anywhere from 1 to 10 minutes and that will reduce the effectiveness of the TDD/BDD cycles, one of the big enemies of workflow is slow workflow where the mind wanders off.

Another option would be to do quick development cycles on the development machine and every once in a while running the tests on the device and lastly ensuring code quality by allowing the CI on the build machine to make sure the code is built and tested on an actual device.

The basic Tenet

All testing should be written in the same way, we prepare code and data for the unit under test, we execute the operation and we should verify the outcome is the one we expected, this is called the AAA Pattern, or Arrange-Act-Assert.

Arrange - prepare objects and data for operation

Act  - execute operation

Assert - check results/side effects

If you have the same arrange for many tests, consider the "setup" phase most testing frameworks have, but to maintain a coherent style one should strive to keep the generic parts in the setup phase and the test specific parts in the test itself.

If you elected to use the "setup" phase, just note that you also have the "tear down" phase to do the cleanup from the setup phase - keeping them out of the test is usually a good practice.

Another tenet is repeatability, tests which are not repeatable cannot be trusted and are soon ignored. avoid testing random values in unit tests.

Unit Tests

Unit tests are the most basic kind of testing but are sometimes misunderstood, the purpose of the unit tests are to verify a needed functionality works in the way it was designed to work during the life of the application, there is a broad interpretation of what constitutes a 'unit' but most agree that a unit is a small chunk of code (or function) that can be accessed from outside the module its in.

Note: while testing every individual functionality manually can also verify the code "works", its not doing it throughout the life of the application, a manual test can lead to a waste of time and will not test the same functionality every time. 

PlatformIO provides a way to run tests and split them into groups, each group can be set to either run or not run on each platform (ESP32, Native and more), we can split by library, header or other significant group.

Project Tests Structure

For very simple projects you can put a single main file in the test folder and run all tests from it, keep in mind if it gets too big it might not fit in ESP32. 

For more complex projects, its possible to create multiple executables in the test folder by using subfolders, each subfolder starting with test_ will create a separate executable.

Like in all other unit test demos, we'll start with a simple calculator, it has addition, subtraction, multiplication and division.

int Calculator::add(int a, int b)
{
    return a + b;
}

int Calculator::sub(int a, int b)
{
    return a - b;
}

int Calculator::mul(int a, int b)
{
    return a * b;
}

int Calculator::div(int a, int b)
{
    return a / b;
}

calculator.cpp

Testing Frameworks

PlatformIO supports out of the box a few testing frameworks, the more popular ones in the embedded world are unity, cpputest and doctest.

Now we'll write some tests

Unity

Unity is popular due to its low overhead and simple use, it was the first framework supported on PlatformIO.

#include <calculator.h>
#include <unity.h> //Unity Testing Framework
#include <runner.h> //Simplifies main()

Calculator calc;

void test_function_calculator_addition(void) {
    TEST_ASSERT_EQUAL(32, calc.add(25, 7));
}

void test_function_calculator_subtraction(void) {
    TEST_ASSERT_EQUAL(20, calc.sub(23, 3));
}

void test_function_calculator_multiplication(void) {
    TEST_ASSERT_EQUAL(50, calc.mul(25, 2));
}

void test_function_calculator_division(void) {
    TEST_ASSERT_EQUAL(32, calc.div(96, 3));
}

void process() {
    UNITY_BEGIN();
    RUN_TEST(test_function_calculator_addition);
    RUN_TEST(test_function_calculator_subtraction);
    RUN_TEST(test_function_calculator_multiplication);
    RUN_TEST(test_function_calculator_division);
    UNITY_END();
}

MAIN(){
    process();
}

test_calculator.cpp

doctest

doctest is a similar framework to Catch, except for the slow compilation time, some developers really like Catch's way of doing things but really hate the slow compilation time, doctest while natively supported by PlatformIO still have issues when compiling for ESP32, you can find a modified doctest in the examples that works fine with both ESP32 and natively.

#define DOCTEST_CONFIG_IMPLEMENT
#define DOCTEST_THREAD_LOCAL
#include <doctest/doctest.h> //doctest testing framework
#include <runner.h> //Simplifies main()

MAIN(){
    const int argc_ = 3;
    const char *argv_[] = {
        "exe",
        "-d",
        "-s"};
    return doctest::Context(argc_, argv_).run();
}

#include <calculator.h>

Calculator calc;


TEST_CASE("calculator addition"){
    CHECK(32== calc.add(25, 7));
}

TEST_CASE("calculator subtraction"){
    CHECK(20 == calc.sub(23, 3));
}

TEST_CASE("calculator multiplication"){
    CHECK(50 ==  calc.mul(25, 2));
}

TEST_CASE("calculator division"){
    CHECK(32 == calc.div(96, 3));
}

test_calculator.cpp

CppUTest

#include <runner.h> //Simplifies main()

#define CPPUTEST_USE_LONG_LONG 1 //mandatory for cpputest to work with esp32
#include "CppUTest/CommandLineTestRunner.h"
#include "CppUTest/TestPlugin.h"
#include "CppUTest/TestRegistry.h"
#include "CppUTestExt/IEEE754ExceptionsPlugin.h"
#include "CppUTestExt/MockSupportPlugin.h"

MAIN(){
    const char * argv_[] = {
        ""
        "",
        "-v",
        "-c",
        "-o",
        "eclipse"
        //"teamcity"//"eclipse"//"junit"
    };
    return CommandLineTestRunner::RunAllTests(5, argv_);
}

#include <calculator.h>

Calculator calc;

TEST_GROUP(Calculator){ };


TEST(Calculator, Addition){
    CHECK(32== calc.add(25, 7));
}

TEST(Calculator, Subtraction){
    CHECK(20 == calc.sub(23, 3));
}

TEST(Calculator, Multiplication){
    CHECK(50 ==  calc.mul(25, 2));
}

TEST(Calculator, Division){
    CHECK(32 == calc.div(96, 3));
}

test_calculator.cpp

If you're migrating to PlatformIO and already use a different framework which is not on the supported list, you might be interested to know how to setup PlatformIO to work with custom framework. 

Environments

To use PlatformIO effectively, we'll need to tell it which setups its going to work with, such as Platforms, Boards and Frameworks, the combination of these can be grouped into environments in platformio.ini, however, environments can contain more than that. 

Lets define two environments, one for ESP32 and one for natively running on the development machine.

[env:esp32]
platform = espressif32
board = esp32doit-devkit-v1
framework = espidf

[env:native]
platform = native

Running Tests

Running tests in PlatformIO is pretty simple, this command will run all tests in all environments, so if we have a native and ESP32 environment defined, it will executes all tests against them.

pio test

But lets say we want to run only native environment:

pio test -e native

Lastly, some labs have the device connected to a remote agent and shared by multiple developers or even a CI agent

pio remote test

Limiting Tests to Specific Environments

We can always use ifdef guards to build tests for specific platforms but for the sake of order we're probably going to want to group common, embedded and desktop tests. I propose that we'll have 3 separate folders for our sample tests and tell PlatformIO to ignore the incompatible tests on the incompatible platforms.

[env:esp32]
platform = espressif32
board = esp32doit-devkit-v1
framework = espidf
test_ignore = test_desktop

[env:native]
platform = native
test_ignore = test_embedded

Now that we have our tests working, lets see how the results look like

>pio test -e native
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 3 tests

Processing test_common in native environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Testing...
test\test_common\test_calculator.cpp:49: test_function_calculator_addition      [PASSED]
test\test_common\test_calculator.cpp:50: test_function_calculator_subtraction   [PASSED]
test\test_common\test_calculator.cpp:51: test_function_calculator_multiplication        [PASSED]
test\test_common\test_calculator.cpp:52: test_function_calculator_division      [PASSED]
------------------------------------------------------------------------------------ native:test_common [PASSED] Took 2.75 seconds ------------------------------------------------------------------------------------

Processing test_desktop in native environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Testing...
test\test_desktop\test_calculator.cpp:49: test_function_calculator_addition     [PASSED]
test\test_desktop\test_calculator.cpp:50: test_function_calculator_subtraction  [PASSED]
test\test_desktop\test_calculator.cpp:51: test_function_calculator_multiplication       [PASSED]
test\test_desktop\test_calculator.cpp:44: test_function_calculator_division: Expected 32 Was 33 [FAILED]
------------------------------------------------------------------------------------ native:test_desktop [FAILED] Took 2.42 seconds ------------------------------------------------------------------------------------

======================================================================================================= SUMMARY =======================================================================================================
Environment    Test          Status    Duration
-------------  ------------  --------  ------------
native         test_common   PASSED    00:00:02.753
native         test_desktop  FAILED    00:00:02.421

_________________________________________________________________________________________________ native:test_desktop _________________________________________________________________________________________________
test\test_desktop\test_calculator.cpp:44:test_function_calculator_division:FAIL: Expected 32 Was 33

Emulators

PlatformIO has integrated a few emulators, unfortunately none of them is for ESP32, however, Espressif has worked on QEMU, which might make it into the supported emulators one day.

Multithreading

Testing multithreaded code does not technically constitutes a unit test, however, sometimes we want to verify the integration between components bridged by concurrency patterns.

More over, unit testing multithreaded code can lead to issues, such as irreproducible or sparse failures with no certain way to check why and in turn lead to tests being ignored or deleted.

In general, when developing multithreaded code the main issues raise when threads share memory and data, depending on architecture and word size, accessing unaligned variables and structs usually compiles to multiple instructions that access and dissect the aligned memory into smaller chunks which will cause issues if multiple threads access that data since the context switch can occur between instructions.

With that being said, if you must test multithreaded code, you'll need to control the timing or wait for something to happen and not depend on sleeps and delays since it will make your code non-deterministic.

If you must share data between threads, do it with the appropriate concurrency pattern, such as messages, queues and lists.

You may find more interesting patterns with etl.

RTOS

RTOS use on a microcontroller can enable higher quality code by separating responsibilities to individual tasks and functions, allowing them to interact safely, the downside of that is the added complexity of working with RTOS. The esp-idf has integrated FreeRTOS already for you, saving some of the learning curve but I do recommend reading the FreeRTOS documentation and even going into some of its source code to understand exactly what's going on, if you really need it, FreeRTOS has been ported to Windows/Linux so you test a part of your code on your development machine.

Continue to static code analysis... 

Tags: esp32, platformio, Testing, Unit Tests