300x250 AD TOP

Search This Blog

Paling Dilihat

Powered by Blogger.

Monday, September 4, 2017

Looking To The IoT Future With PlatformIO And ESP32

Looking To The Future With PlatformIO And ESP32 or Why I Think the ESP32+PlatformIO is a game changer.

I've started doing electronics and other technical stuff since I was very little but I've left this hobby in the late 90's. Since then it all went under the radar for me, I would still buy the occasional Elektor magazine or browse the kits section in an electronics shop but it lost my serious attention.

About 10 years ago I've wanted to try building something and so bought the immortal Hakko 936 and a Power Supply from CircuitSpecialists and later made an order of a few transistors and ICs from DigiKey but that project did not really work as I forgot too much and needed to learn too much but without the proper books and it caused me to lose interest again.

At 2012 I've had another idea how to build an all budget house automation system and so found myself browsing AliExpress and it rekindled my love for soldering smells and building things.

So why am I telling you this?

Around that time the Arduino ecosystem started flourishing and everyone could start doing microprocessor work easily and at home and although my first microprocessor was a Microchip PIC, it was around 2005, the IDE sucked, you could do either assembler only or limited C code, I've lost interest pretty fast and I bought about 3 processors for about $25...

MPLAB 6.10
Attractive eh?
At the time assembly was understandable, the thing had about 1-8kb for code and while you can write optimized C code or the compiler can optimize for code size, assembly was the normal thing to do.

So again, why am I telling you this?

Like Steve Ballmer immortal words "Developers Developers Developers!", a good development environment is one of the primary reasons a product can get popular.

While Arduino IDE is a perfect for beginners, you quickly reach the limit of what you can do without a modern IDE, such as Visual Studio, Eclipse or any real code parsing, autocompletion, quickinfo IDE, the fun part is discovering that Atmel AVR (the processor many Arduinos are based on) already has an IDE, Atmel Studio.

Its even better to find out someone has already made the Arduino IDE compatible with Visual Studio IDE, which began to be my favorite since Visual Studio 2013 when the C++ parser was improved tremendously, that plugin is Visual Micro.


But lately I've found myself limited with what I can do in Visual Micro. I've started programming the ESP32, which is my favorite microprocessor at the moment. The more I learn about it, the more I realize this is an enterprise grade, industry 4.0, IIoT, whichever buzzword you want to stick on it, it can probably do it with its dual core, 520kb memory, 4MB flash, Wifi, BT, 18 channel ADC, 4SPI, 2 I2C, 3 UARTs CAN bus, IR, PWM, sleep modes, encryption and cryptographic acceleration. A thing of beauty for $5 module or $8 development board with USB Programmer.

So why am I telling you this?

I've recently decided to take another look at PlatformIO, while their github started at about 2014, I'm only taking interest now since it can probably do what I want it to do, give me access to esp-idf with a decent editor and make the noise go away until I need it (build environment).

I'll be blunt, the IDE is still rough around the edges, Atom is not as fun as Visual Studio, VS Code does not have a C++ parser as good as Visual Studio's and overall development experience is about 7 out of 10 but that thing does its magic and enables me to explore the entire FreeRTOS and esp-idf framework so I'll bear the quirks.

In any case, I've installed the plugin about a week ago, since then I've patched and did a pull request for allowing custom partitions and I've even found a way to measure task CPU usage in FreeRTOS which eluded me in-spite of all my efforts in Visual Micro/Arduino.

Do I recommend it?

Only if you've surpassed the capabilities Visual Micro/Arduino gave you since there's still no replacement for Visual Studio C++ parser in my opinion. and Yes, I did try Eclipse some time ago but was not satisfied with the results so I've abandoned it.

But wait, there's good news, I've found a way to use Visual Studio with PlatformIO. At first I saw issue 543, it did not work so well for me, so I combined it with my own solution and it worked, I now have a project that works with PlatformIO! :-)

PlatformIO Use

Lets continue working on an example of what I could do with PlatformIO that I couldn't do with Visual Micro/Arduino, first lets discuss how projects are built in PlatformIO as far as I could tell in this early stage.

With PlatformIO the do-it-all command is... you guessed it - platformio (or platformio.exe in windows) or pio.

In our case, for esp32 with esp-idf:
platformio init --board esp32dev --project-option "framework=espidf"

Now the project is ready for development, you can open it with VSCode or Atom and start working.

But there are two enhancements we can do right now, first we should edit platformio.ini and add the following line in the end of env:esp32dev so the "platformio device monitor" will not crap out:
monitor_baud = 115200

Then we can create our Visual Studio project (not vscode!):
platformio init --ide visualstudio

But we're not done yet, Visual Studio intellisense is designed to work with Microsoft Visual C++, we'll need to add a file and set it in Project Properties -> NMake -> Forced Includes

At first I've attempted to use the solution in Issue 543, but it did not deliver, so I've modified it to this but its a work in progress, so most of the intellisense is working but there are still some quirks.

#ifndef __GNUC__
#define __GNUC__ 2

#ifndef __STDC__
#define __STDC__

#ifdef _WIN32
#define __attribute__(A) /* do nothing */

#ifdef _MSC_VER#define __asm__(x)
#define __extension__(x)
#define __attribute__(x)
#define __builtin_va_list int
#define __extension__
#define __inline__
#define __builtin_constant_p
#define _Bool bool
typedef int __INTPTR_TYPE__ ;

typedef unsigned int __UINTPTR_TYPE__;

So now that we have a good environment, we can start working.

Custom Partition Table

One of the first features I wanted to try and implement with PlatformIO which I couldn't with Visual Micro is a custom partition table. We add a new partitions_table.csv and follow the instructions.

You will also need to modify sdkconfig.h and update these defines:
#define CONFIG_PARTITION_TABLE_FILENAME "coredump_partitions.csv"
#define CONFIG_PARTITION_TABLE_CUSTOM_FILENAME "coredump_partitions.csv"

In this case I wanted to try a partition table with ota, a fat filesystem and a coredump.

# Name,   Type, SubType, Offset,   Size
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
eeprom,   data, 0x99,    0x290000,0x1000,
fat,      data, fat,     0x291000,0x15F000,
coredump, data, coredump,0x3F0000,0x10000

I've wrote a small snippet of code to display the partition table once the esp boots up and now we can upload the code to esp32:

platformio run -t upload

If we're successful, we'll see:

Hard resetting...

==== [SUCCESS] Took 20.79 seconds ====

Now we can monitor esp32:

platformio device monitor

And see the results:

- label           type/subtype   address:size            encrypted
- app0            0x00/0x10      0x00010000:0x00140000   0
- app1            0x00/0x11      0x00150000:0x00140000   0
- nvs             0x01/0x02      0x00009000:0x00005000   0
- otadata         0x01/0x00      0x0000e000:0x00002000   0
- eeprom          0x01/0x99      0x00290000:0x00001000   0
- fat             0x01/0x81      0x00291000:0x0015f000   0
- coredump        0x01/0x03      0x003f0000:0x00010000   0

Great! First success with the esp-idf and PlatformIO!

You can find the sources for this POC here:

Core Dumps

One of the features that an enterprise grade product needs is to be able to analyze crashes, not just the ones that a developer is able to attach a debugger to or the ones that can be reproduced but the ones that seldom happen, the ones that happen only in production, the hard ones.

To be able to do that, Alexey Gerenkov implemented saving the core dumps to a special flash partition, but I wanted to do something a bit different, I wanted to send the core dumps back to my server for monitoring, so whenever the device would start it will go to the core dump partition, check if it not empty, then be able to send it to the server, save it to the SPI fat file system or anything else that might be needed.


I had a good starting point for a crash, I've used this code.

You need to select a partition table with a core dump partition, this is where the previous POC came into play as the PlatformIO esp-idf framework can only use partitions_singleapp.csv as far as I can tell without my modification.

I then started to look for a way to retrieve the core dump partition, you will need to use

I've then used esp_partition_mmap to easily access the partition and check its length.

I think it might be better to check for the start and end markers instead of a "not empty" (0xFF) characte, but its good enough for the POC.

So now we have the start address and length of the core dump, we can do whatever we want with it, dump it to screen as base64, there's also a built in function that does that on crash, but you need to configure it:

Again, this is not what I wanted as there will be no one connected to the device to grab that core dump from the console.

Eventually we can assume we have a core dump, make sure you're uploading the core dump along with the firmware version it happened in otherwise you won't know where it crashed and you'll get the wrong results.

We can now run the command to get the core dump in text form:
espcoredump.py info_corefile -t b64 -c test_crash.txt -g toolchain-xtensa32\bin\xtensa-esp32-elf-gdb.exe ".pioenvs\esp32dev\firmware.elf" 

And the result:
==================== ESP32 CORE DUMP START ====================

================== CURRENT THREAD REGISTERS ===================
pc             0x400d0901       0x400d0901 <recur_func()+73>
lbeg           0x400014fd       1073747197
lend           0x4000150d       1073747213
lcount         0xffffffff       4294967295
sar            0x0      0
ps             0x60f20  397088
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x400d08e0       1074596064
a1             0x3ffc7f20       1073512224
a2             0x2      2
a3             0x3f403578       1061172600
a4             0x3ffc7f60       1073512288
a5             0x3ffc271c       1073489692
a6             0x0      0
a7             0x0      0
a8             0x5      5
a9             0xffffffad       -83
a10            0x20     32
a11            0x3ffc656c       1073505644
a12            0x1      1
a13            0x80     128
a14            0x1      1
a15            0x0      0

==================== CURRENT THREAD STACK =====================
#0  0x400d0901 in recur_func () at src\ est_core_dump.cpp:80
#1  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#2  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#3  0x400d0924 in unaligned_ptr_task (pvParameter=0x0) at src\  est_core_dump.cpp:90

======================== THREADS INFO =========================
  Id   Target Id         Frame
  9    process 8         xQueueGenericReceive (xQueue=0x3ffc2ecc, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  8    process 7         xQueueGenericReceive (xQueue=0x3ffc2904, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  7    process 6         prvTimerTask (pvParameters=0x0) at .platformio\\packages\\framework-espidf\\components\\freertos\     imers.c:445
  6    process 5         failed_assert_task (pvParameter=0x0) at src\   est_core_dump.cpp:103
  5    process 4         0x400d0c12 in init_analysis_task (pvParameter=<optimized out>) at src\ est_core_dump.cpp:130
  4    process 3         bad_ptr_task (pvParameter=0x0) at src\ est_core_dump.cpp:51
  3    process 2         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
  2    process 1         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
* 1    <main task>       0x400d0901 in recur_func () at src\    est_core_dump.cpp:80

======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x400c0000 0x0 RW
.iram0.vectors 0x40080000 0x400 R XA
.iram0.text 0x40080400 0x7860 R XA
.dram0.data 0x3ffc0000 0x1d20 RW A
.flash.rodata 0x3f400010 0x3e64 RW A
.flash.text 0x400d0018 0xed20 R XA
.coredump.tasks 0x3ffc6504 0x164 RW
.coredump.tasks 0x3ffc7e60 0x1d4 RW
.coredump.tasks 0x3ffc559c 0x164 RW
.coredump.tasks 0x3ffc5430 0x160 RW
.coredump.tasks 0x3ffc5028 0x164 RW
.coredump.tasks 0x3ffc4eb0 0x16c RW
.coredump.tasks 0x3ffc6398 0x164 RW
.coredump.tasks 0x3ffc76e0 0x14c RW
.coredump.tasks 0x3ffc622c 0x164 RW
.coredump.tasks 0x3ffc6ed0 0x154 RW
.coredump.tasks 0x3ffc8848 0x164 RW
.coredump.tasks 0x3ffc86f0 0x14c RW
.coredump.tasks 0x3ffc6004 0x164 RW
.coredump.tasks 0x3ffc5ea0 0x158 RW
.coredump.tasks 0x3ffc2d60 0x164 RW
.coredump.tasks 0x3ffc2bd0 0x184 RW
.coredump.tasks 0x3ffc3328 0x164 RW
.coredump.tasks 0x3ffc3190 0x18c RW

===================== ESP32 CORE DUMP END =====================

CPU Utilization

This is a bit of a hack, since I would rather not change esp-idf code without commiting the change back to the main repo, I've tried many workarounds, eventually the last nail in the coffin was the fact there's a definition for portGET_RUN_TIME_COUNTER_VALUE in freertos/portmacro.h, making it impossible to override from outside the framework.

So.... I had to modify portmacro.h, a change I plan to ask to push back into esp-idf:

#define portGET_RUN_TIME_COUNTER_VALUE()  xthal_get_ccount()


Then I needed to add the functions for initializing the counter and getting the counter values in sdkconfig.h:

#ifndef __ASSEMBLER__
#ifdef __cplusplus
extern "C"{

#ifdef ESP32
extern unsigned long get_run_time_counter_value();
extern void init_run_time_counter();

#ifdef __cplusplus

#define portGET_RUN_TIME_COUNTER_VALUE() get_run_time_counter_value()
#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() init_run_time_counter()


#define configUSE_TRACE_FACILITY 1

(In case you're wondering about the #ifndef __ASSEMBLER__ its because the sdkconfig.h is included in .S files)

I then went ahead and used system_get_time to get the value for the counter as it was the only stable one for now, the others are core dependent so every time a task would switch core due to context switching, the counter value will be different showing skewed results.

I know system_get_time is deprecated but I've yet to find a better function with stable results that won't crash, I've tried gettimeofday but it crashed the device.

In any case, this should not be used in production software according to FreeRTOS.

After running for a few cycles, we get these results:

Tasks currently running:
Printing Stats:
high_load1      502167712        49
medium_load2    15551453         1
display_stats   235900          <1
medium_load1    18987019         1
low_load2       384767          <1
low_load1       422990          <1
IDLE            34476           <1
IDLE            977132          <1
high_load2      505107417        50
Tmr Svc         60              <1
ipc0            340010          <1
ipc1            22441           <1

iterations low: 70 medium: 3156 high: 100096

What I would like you to notice is the amount of iterations each function executed, bear in mind this is an RTOS, so higher priority tasks run first, if the high and medium tasks are not in a wait section such as vTaskDelay, the low priority won't get any time, taskYIELD might not be enough here so make sure you don't have high priority tasks that never release CPU time.


This was a short demonstration of a few things the ESP32 can do in terms of enterprise grade devices, the more I read esp-idf code, the more I realize its potential. 

I think this development board is one of the most fun ones I played with, in the upcoming weeks I'll see if its possible to apply enterprise grade architecture and design with it, I've already implemented a lean dependency injection for it and did a POC controlling it with Azure IoT Hub. 

You must have seen that:

In my opinion ESP32 + FreeRTOS + Azure IoT Hub + PlatformIO is overlooked and has the potential of realizing many Industry 4.0 projects.

To understand the potential, this is a small section from Microsoft Build 2017 First day Keynotes, you will notice what analyzing seemingly naive data can give you in the long run a change in power consumption, vibrations, temperatures can all point to an imminent failure, predicting maintenance, short product life and much much more.

I think the next best step will be to learn about PlatformIO unit testing capabilities.

I'm looking forward to see what espressif will think of next, just imagine the PC we had about 20 years ago is now the size of a coin and can be programmed by anyone.

Tags: , , ,


Post a Comment