300x250 AD TOP

Search This Blog

Pages

Paling Dilihat

Powered by Blogger.

Sunday, September 24, 2017

ESP32 as Professional Grade/Industry 4.0 Device

According to Forbes, IoT is a $264 Billion market while Industry 4.0 will be $152 billion by 2022, these predictions can lead us to believe in IoT as more than a fleeting trend and should prepare us for the future to come where embedded developers will be plucked from other software development fields, as micro-controllers get more powerful they can also host a more modern programming paradigms and we’ll explore some of these for the sake of making embedded development more robust as well as faster time to market development.



Many words have been written about the advantages and the future that Industry 4.0 devices can achieve for manufacturing, factories, plants and countless ideas have been in the open, hackathons have been performed to find the next killer device, services and applications. This article is not about that, its about how to achieve the fastest way to develop such applications for wide scale deployment.

Industry 4.0 plays a significant role in industrial optimization, by monitoring systems and analyzing the sensor inputs either by statistics or by learning algorithms, failure prediction, load analysis and and optimizations can be achieved. In turn cost cutting and growth are two byproducts that every business needs.

While some of the big time players offer ready made solutions, all the parts have been in the open source community for quite a while, IoT no longer a toy for the geeks but a growing movement to improve a person's life and can be applied in industrial setting.

In my humble opinion, IoT and Industry 4.0 are not there to take jobs but to take the mundane and boring work and allow a better working condition, more interesting and less erosion and by that increase engagement and productivity.


Industrial Usage


While the ESP32 is not a replacement for commercial PLC/SCADA devices, it can achieve that goal pretty fast, sensors and relay modules or SSR modules can make connectivity relatively easy task and if you’re missing a few inputs or outputs, a multiplexer can be a solution as long as your switching speed is within the tolerances.

But what about software?

https://github.com/node-red/node-red


At this moment, node-red is the most popular IoT control software, it is very easy to use and many tutorials have been done in both video and articles. Other options are Eclipse Kura and Project Flogo.

If your needs are more industrial, a few SCADA open source project:

http://rapidscada.org

RapidSCADA - Rapid SCADA is free, open source, full featured SCADA software.

Eclipse NeoSCADA™ is flexible. It is not an out of the box solution, but a set of tools that can be combined in many different ways. It provides development libraries, interface applications, mass configuration tools, front-end and back-end applications.

IndigoSCADA - A small footprint SCADA system entirely developed in C and C++ with multiple OS support and multiple front end protocol drivers.

ScadaBR is free, open source free software for the development of Automation, Data Acquisition and Supervisory Control applications.

SZARP is a full-featured SCADA system designed for monitoring slowly changing industrial processes, for example operations of urban heating systems. It is completely free software, published under terms of the GNU General Public License 2.0 (or any later version).


Why ESP32

While all of the ideas in this article are common to most of the IoT devices with a decent hardware, this article is about ESP32, which is even more suitable than most of the other micro-controllers I’ve encountered lately.

While Espressif is well known in the hobbyist community, it is less used at the professional and industrial sectors, in my opinion not rightfully so.

Espressif ESP32 gives a fair fight against many micro-controllers. Maxim, Texas Instruments, Microchip, Nordic Semiconductor, NXP, Silicon Labs, ST and Atmel all provide microprocessors that can compete, some more popular than others, some provide a free IDE, some provide very expensive SDKs and most of them has a sort of RTOS. What Espressif did right is opening up their SDK and development environments so others like PlatformIO can make development for their hardware accessible, their choice of FreeRTOS allowed them to open source the entire framework and by that help anyone willing to learn about embedded development to get the information, details and insights that only a source code can provide.

Around accessibility to hardware and software a community can be created and in turn the hardware can get support, software, components and drivers like no other, essentially by giving the community esp-idf, design documents and specs, this gift keeps on giving back, volunteers contribute back their time and effort while Epressif can sit, relax and plan their next killer chips.

To give a good example for how not to push a product we can take Intel Curie, Galileo, Joule and Edison, while Intel did make an effort to push their maker chips into the market, most of them were way overpriced and under-performing and in spite of their efforts by giving many many of these devices as gifts, prizes, participating in hackathons and more, their downfall was the lack of documentation and community support. The companies that did try to build a business around Intel maker chips are now left hanging and need to redesign their products.

https://www.cnx-software.com/


ESP32 Ecosystem

Some of the components that compose the esp32 ecosystem are:

esp32 microprocessor - A dual core 240Mhz microprocessor, 520kb ram, RTC, ULP, 34 GPIOs, network connectivity and multiple UARTs, I2Cs, I2Ss, SPIs, CANBUS, ADCs and DACs with external SPI EEPROM that can be from 1-16MB.

esp-idf – Espressif made a great effort to build a Software Development Kit (SDK) which exposes most if not all of the hardware’s capabilities.

Networking - WiFi/BT/Ethernet connectivity, lwIP TCP/IP Stack.

Security/Cryptography – Encryption acceleration instructions (AES (128, 192, 256) / SHA (1,256,384,512) Acceleration), Networking encryption, Flash encryption, Secure Boot and JTAG Disabling. As well as mbedtls library (formally known as PolarSSL), OpenSSL API translation and libsodium.

Cloud Ready – both AWS IoT and Azure IoT components exists and tested.

FreeRTOS – A full implementation including Tasks and real-time task scheduler, Timers, Queues, Interrupt handling, critical sections and events.

Modern C++ 11/14 GCC Compiler – Modern C++11 and up offers many advantages over C++98, among them is increased productivity, speed, stability, safer code and portability are the right buzzwords but smart pointers, improved collections, lambdas and threads and more are the most common advantages anyone should start looking into.

Industrial/Professional Grade

By combining all these components, Espressif made sure that esp32 will fit many applications, including industrial and professional grade, but what does it mean to be able to write industrial or professional grade applications?

First, lets define industrial and professional grade applications.

Industrial/Professional Hobbist
Paid by corporations/factories  Paid by private consumers/Free/Open Source
Multiple teams/developers are working for a common goal One or more developers are working on their spare time or with very little budget
Mission-Critical Systems, essential for a business to grow Occasional downtime is expected, nothing extreme will happen if the system is down.
Reliable, Robust application, should prove business value Reliability/Robustness is not guaranteed.
Fault Tolerant/High Availability/Recoverability Repair/Support if any, can be at business hours
Predictable/Repeatable Performance and Timing Performance is not defined and may not have a negative effect
Multi Layered, Discreet Components, Separation of concerns Boundaries are not clear, programming paradigms are flexible.
Queued Work, Interrupt Handling Slow Response and inconsistent results might be acceptable.
Connectivity Essential Connectivity Optional
Higher Costs Medium Costs

Lets get things straight, industrial/enterprise applications do not mean higher quality of code and hobbyist software does not mean lower quality, all it means is that they both have different requirements, for example, a hobbyist application developer might spend more time on user experience and UI, a commercial developer can and probably will spend more time on stability, load testing and making sure the software won't fail when a 30,000 RPM drill is coming at an aluminum block at a very fast speed. different requirements, that's all. More over, these days the boundaries between industrial and hobbyist products are getting more and more fuzzy because there is more awareness.

Now, lets imagine you need to write a program that almost never fail, imagine you need it to be up and running all the time, imagine you need to know what its doing, how its doing, its health, what happened to it when it failed, you need to update it when you find a new problem or find a more efficient way to do something, imagine its as secure as possible.

“a program that almost never fail”

While it is a nice dream, software does fail. But what can we do to solve the most common failures and improve stability?

  • Use Modern Compilers and programming paradigms, object oriented development, RAII, smart pointers, collections, lambdas, anything that can help you get your job done in a safer manner. Raw pointers have been around since C, but smart pointers are safer for use. Prefer dynamic casts and static casts over reinterpret cast and c casts.
  • Tests are essential to a stable application, while running tests for xtensa processors on a PC inside QEMU could get complicated, you can use abstraction layers and mocks to run and test most of your code on your development OS, esp-idf and PlatformIO have testing facilities to ease the pain. Also, as the code is a standard C++ code, you can use cppcheck for static code analysis.
  • Always use source control, even for POCs.
  • Build Automation, Versioning.
  • Crash Dump Analysis/Stack Overflow Detection are part of esp-idf, use them.
  • Logs are important for diagnosing a problem or a program misbehave, while esp-idf provides a logging mechanism which can be used for tracing, its also good to be able to save these messages to files and later transmit for analysis and post mortem or use a network syslog.
  • Enable OTA updates, use versioning, certificate signing and a staging environment that contains all the hardware this firmware will be deployed.


What can we do to make the software more reliable? robust? secure?

  • Input Validation - always verify your inputs, buffer overrun is still one of the major problem that leads to hacking, SQL injection is still not dead.
  • Tests - test your code with invalid input, long WiFi names, external hardware glitches. disconnect power, reconnect power. worse case errors should lead to crashes, not compromise security.
  • Use more advanced paradigms, like queues, events, callbacks, exceptions, micro components, layers, inversion of control, control weaving.
  • Logs should have everything you need to diagnose a problem from remote, you shouldn't fly someone across an ocean just to find out a configuration problem.


What can we do to make the software fault tolerant, highly available and recoverable?

  • Fail Fast - failing fast leads to development finding the problem at development time, never swallow errors, if you must, log them first.
  • Exception Handling - an exception happens at an exceptional state, it should not happen, if it did, find out why. don't use exceptions for non-exceptional errors.
  • Reboot upon failure - esp-idf is configured to reboot on panic, invalid pointers access will lead to reboots, . Stack overflow leads to a reboot.
  • Watchdog - sometimes failure does not mean a crash, sometimes it means that a software no long responds at the designed time, sometimes a task enters an infinite loop and waste all its time on doing nothing. watchdogs and heartbeats are designed to detect these situations, use them.
  • High CPU usage detection - while not part of FreeRTOS failures, if you see a poorly performing software, add it, it's not hard. 
  • Brownout detection - brownout is a state of too low power for the hardware to work properly, the system should reboot on brownout detection. Fault injection could be just another way to hack the hardware.
  • If appropriate, use more than 1 device, perform election to select the better device, if it fails, failover to the redundant device, alternatively use heartbeat to select the master device. An example usage would be a redundant controller on a CCR breathing system.


What can we do the to make the software more predictable? High performance? Accurate Timing?

  • RTOS - RTOSs provide multi tasking capabilities, queues, interrupt handling in a safe way.
  • Multicore, faster microcontroller - if your core is busy all the time, it won't be able to react to changes in time, leading to reduced performance in the best case and dependent hardware crash/accident in the worse case.
  • Use Interrupts and Timers - its the age old Polling vs Push.


A Technical Drill Down

esp-idf is a very interesting SDK, Espressif made a huge effort to make it as easy to use as possible, you can find APIs controlling the hardware, drivers, pthread compatible APIs and C++ stdlib, even glob, regex and tar and miniz, all of this makes programming for the ESP32 relatively easy with many features that makes porting existing code a breeze.

esp-idf includes a few interesting features and components that aids building a more robust application, among them is SPI filesystems, both FAT and SPIFFS (spoiler: FAT is faster), you can use the filesystems to store log and data files, the SPI EEPROM include a wear leveling layer which helps to prolong the amount of writes possible. Settings can be stored in NVS so formatting the filessytem won't affect it, lastly if you need more than the SPI memory, you can always use the SD-Card Library.

Cloud Connectivity is essential to large scale monitoring and control, esp-idf includes AWS IoT connectivity and Microsoft Azure IoT connectivity works as well, TelegramBot API, CoAP and MQTT are available as well.

lwIP is a IPv4/IPv6 TCP/IP Stack which have a few interesting features, among them is DNS/mDNS resolution, SNMP, DHCP, AUTOIP, PPP and L2TP , all of them are the basics of networking, among the things you can do with it are DHCP servers, SNTP, PING, HTTP(s) Client/Server, CoAp Client/Server, OpenSSL Client/Server.

nghttp2 is an http2 client/server.

mbedTLS/libsodium are cryptographic libraries, using them is mostly preference, both are very similar, and have their own set of features and quirks, you should make an informed decision by researching them.

FreeRTOS APIs include Task management, including a scheduler and monitoring, data structrures that incude Queues/Stacks, Ring Buffer, Locking Mechanisms like Semaphores and Mutexes. Interrupt Aware APIs

But free is not always good, free means you can use it without paying but sometimes you need it with support, sometimes you need it with a guarantee, FreeRTOS have a commercial alternative called OpenRTOS, it’s the exact same RTOS just with a different licensing. Another Alternative is the SAFERTOS. For more information you can see a comparison here.

Power Management is an important features for low power applications, the ESP32 have multiple sleep levels with an ULP processor that can reduce power requirements even further.

Debugging – while personally I don’t believe in straight up debugging as its usually reducing the quality of logging and on production you can mostly work with logs and crash dumps, Espressif got OpenOCD with JTAG working. esp-idf comes with a unit test app based on Unity and  If you choose to use PlatformIO, you can use its testing facilities as well.

RTOS

Other RTOSs are not bad, on the contrary, but SDKs have a very steep learning curve, while getting an example up and running could take anywhere from 5 minutes to a few hours, being proficient in that SDK takes time, learning its quirks and overcoming its limitations takes study, but if you take a look at other RTOSs , most of them have almost the same structure and APIs.

But why use a popular RTOS? What does it mean?
Learning about a device from its datasheet could take a long time, especially those 100 pages datasheets for complex devices, on the other hand, using a library that is already working and tested can take significantly less time.

One of the advantages of being a part of the Arduino community is that its a huge community, so many developers already read the datasheet and wrote drivers and sample codes, some of them are very low quality but like every kind of community, some are very good.

ESP32 can be used two additional RTOS that I know of:

Simba http://simba-os.readthedocs.io

NuttX http://www.nuttx.org/

For more information, you can look at this comparison.

Licensing

While the ESP32 SDK is Apache License, always check license for other components and libraries when you use them. FreeRTOS have a different license, some LCD display drivers have another etc' etc'

Always consult a license lawyer when using for commercial applications

Rapid Prototyping

SDKs are a great way to explore the device’s capabilities but although C++ these days is very easy to learn, sometimes the developers who do the proof of concept and the exploration come from a different field, electronics, mathematics, physics and biology to name a few and C++ might not be their preferred language, a few other languages have been implemented on the ESP32

- LUA - Lua RTOS is a real-time operating system designed to run on embedded systems, with minimal requirements of FLASH and RAM memory.
- MicroPython - aims to put an implementation of Python 3.x on microcontrollers and small embedded systems.
- Basic - Basic interpreter hidden in esp32 silicon

A Few Interesting Projects

Web Radio - MP3 web radio project.

Camera Demo - With OV7725 camera module.

Alexa Voice Service (AVS) client for ESP32- https://github.com/GOLDELEC/Huan

WebSocket example project - https://github.com/ThomasBarth/WebSockets-on-the-ESP32

WiFiManager - WiFi Connection manager with fallback web configuration portal

libesphttpd is a HTTP server library. It supports integration in projects running under the non-os and FreeRTOS-based SDK. Its core is clean and small, but it provides an extensible architecture with plugins to handle a flash-based compressed read-only filesystem for static files, a tiny template engine, websockets, a captive portal, and more.

CAN Bus Driver

Low Powerdeep-sleep functionality if the ESP32

Console component - create an interactive shell.

ESP32 QEMU - https://github.com/Ebiroll/qemu_esp32/

ADNS3080 - Is a mouse optical flow, while you can find example code at ArduPilot, I couldn't find a source code but its a standard SPI and should be relatively easy to port, here's an example use.

Ethernet Connection - Using LAN8720

Development Environment

Eclipse https://exploreembedded.com/wiki/Setting_up_Eclipse_for_ESP-IDF

VSCode http://docs.platformio.org/en/latest/ide/vscode.html

Visual Studio http://docs.platformio.org/en/latest/ide/visualstudio.html

ATOM http://docs.platformio.org/en/latest/ide/atom.html

ESP32 Books, Tutorials and Samples

Neil Kolban has invested a lot of time writing a book, samples and tutorials and should get a special thank you due to the amount of effort he made.

https://github.com/nkolban/esp32-snippets

https://leanpub.com/kolban-ESP32 (https://www.robolinkmarket.com/datasheet/kolban-ESP32.pdf)

https://www.youtube.com/channel/UChKn_BlaVrMrhEquPNI6HuQ

Espressif Technical Reference: http://espressif.com/sites/default/files/documentation/esp32_technical_reference_manual_en.pdf

Development Kits

ESP-WROVER-KIT looks very interesting at $47.00, comes with external 4GB RAM, LCD and JTAG .

ESP32 is also available in many flavors and configurations on the cheap:

A Kit, In a simple breadboard form, An alternative breadboard form, With OLED, In an easy way to change the SPI EEPROM, or this.

In theory you can use this chip to expand the EEPROM to 16MB (untested).

With a Battery, With a LoRa module and OLED displayWith Soil, Temperature and humidity sensorsBattery and display in a compact form.

If you need a Flashing Fixture.

Further Reading and References

You can always explore other options and find the best suitable for your needs, you can find a list here (http://www.softwareforenterprise.us/list-of-free-and-open-source-scada-m2m-systems/)


Summary

In my opinion 3 things drive the Arduino community, cost, accessibility and open source. While an Enterprise or a Factory might not care so much to spend $10,000 - $15,000 on a development environment, they will start caring when they can easily find software developers that already know the hardware being used, in financial terms, the cost of training a new employee are far greater than their hardware.

While I did not bring any real world examples of using ESP32 for industrial purposes, I think I did bring a lot of information on how to do it, the future is inevitable, many software developers are not familiar with hardware but their knowledge and expertise in an professional software practices is mostly far greater than the average hardware programmer, but different requirements will make that previously clear boundary a thing of the past....
Tags: , ,

Monday, September 4, 2017

Looking To The IoT Future With PlatformIO And ESP32

Looking To The Future With PlatformIO And ESP32 or Why I Think the ESP32+PlatformIO is a game changer.

I've started doing electronics and other technical stuff since I was very little but I've left this hobby in the late 90's. Since then it all went under the radar for me, I would still buy the occasional Elektor magazine or browse the kits section in an electronics shop but it lost my serious attention.




About 10 years ago I've wanted to try building something and so bought the immortal Hakko 936 and a Power Supply from CircuitSpecialists and later made an order of a few transistors and ICs from DigiKey but that project did not really work as I forgot too much and needed to learn too much but without the proper books and it caused me to lose interest again.

At 2012 I've had another idea how to build an all budget house automation system and so found myself browsing AliExpress and it rekindled my love for soldering smells and building things.

So why am I telling you this?

Around that time the Arduino ecosystem started flourishing and everyone could start doing microprocessor work easily and at home and although my first microprocessor was a Microchip PIC, it was around 2005, the IDE sucked, you could do either assembler only or limited C code, I've lost interest pretty fast and I bought about 3 processors for about $25...


MPLAB 6.10
Attractive eh?
At the time assembly was understandable, the thing had about 1-8kb for code and while you can write optimized C code or the compiler can optimize for code size, assembly was the normal thing to do.

So again, why am I telling you this?

Like Steve Ballmer immortal words "Developers Developers Developers!", a good development environment is one of the primary reasons a product can get popular.

While Arduino IDE is a perfect for beginners, you quickly reach the limit of what you can do without a modern IDE, such as Visual Studio, Eclipse or any real code parsing, autocompletion, quickinfo IDE, the fun part is discovering that Atmel AVR (the processor many Arduinos are based on) already has an IDE, Atmel Studio.

Its even better to find out someone has already made the Arduino IDE compatible with Visual Studio IDE, which began to be my favorite since Visual Studio 2013 when the C++ parser was improved tremendously, that plugin is Visual Micro.


http://www.visualmicro.com/forums/YaBB.pl?num=1462975225/10

But lately I've found myself limited with what I can do in Visual Micro. I've started programming the ESP32, which is my favorite microprocessor at the moment. The more I learn about it, the more I realize this is an enterprise grade, industry 4.0, IIoT, whichever buzzword you want to stick on it, it can probably do it with its dual core, 520kb memory, 4MB flash, Wifi, BT, 18 channel ADC, 4SPI, 2 I2C, 3 UARTs CAN bus, IR, PWM, sleep modes, encryption and cryptographic acceleration. A thing of beauty for $5 module or $8 development board with USB Programmer.




So why am I telling you this?

I've recently decided to take another look at PlatformIO, while their github started at about 2014, I'm only taking interest now since it can probably do what I want it to do, give me access to esp-idf with a decent editor and make the noise go away until I need it (build environment).

I'll be blunt, the IDE is still rough around the edges, Atom is not as fun as Visual Studio, VS Code does not have a C++ parser as good as Visual Studio's and overall development experience is about 7 out of 10 but that thing does its magic and enables me to explore the entire FreeRTOS and esp-idf framework so I'll bear the quirks.

In any case, I've installed the plugin about a week ago, since then I've patched and did a pull request for allowing custom partitions and I've even found a way to measure task CPU usage in FreeRTOS which eluded me in-spite of all my efforts in Visual Micro/Arduino.

Do I recommend it?

Only if you've surpassed the capabilities Visual Micro/Arduino gave you since there's still no replacement for Visual Studio C++ parser in my opinion. and Yes, I did try Eclipse some time ago but was not satisfied with the results so I've abandoned it.

But wait, there's good news, I've found a way to use Visual Studio with PlatformIO. At first I saw issue 543, it did not work so well for me, so I combined it with my own solution and it worked, I now have a project that works with PlatformIO! :-)


PlatformIO Use

Lets continue working on an example of what I could do with PlatformIO that I couldn't do with Visual Micro/Arduino, first lets discuss how projects are built in PlatformIO as far as I could tell in this early stage.

With PlatformIO the do-it-all command is... you guessed it - platformio (or platformio.exe in windows) or pio.

In our case, for esp32 with esp-idf:
platformio init --board esp32dev --project-option "framework=espidf"

Now the project is ready for development, you can open it with VSCode or Atom and start working.

But there are two enhancements we can do right now, first we should edit platformio.ini and add the following line in the end of env:esp32dev so the "platformio device monitor" will not crap out:
monitor_baud = 115200

Then we can create our Visual Studio project (not vscode!):
platformio init --ide visualstudio

But we're not done yet, Visual Studio intellisense is designed to work with Microsoft Visual C++, we'll need to add a file and set it in Project Properties -> NMake -> Forced Includes




At first I've attempted to use the solution in Issue 543, but it did not deliver, so I've modified it to this but its a work in progress, so most of the intellisense is working but there are still some quirks.

#define _ALLOW_KEYWORD_MACROS
#ifndef __GNUC__
#define __GNUC__ 2
#endif

#ifndef __STDC__
#define __STDC__
#endif

#ifdef _WIN32
#define __attribute__(A) /* do nothing */
#endif

#ifdef _MSC_VER#define __asm__(x)
#define __extension__(x)
#define __attribute__(x)
#define __builtin_va_list int
#define __extension__
#define __inline__
#define __builtin_constant_p
#define _Bool bool
typedef int __INTPTR_TYPE__ ;

typedef unsigned int __UINTPTR_TYPE__;
#endif

So now that we have a good environment, we can start working.


Custom Partition Table

One of the first features I wanted to try and implement with PlatformIO which I couldn't with Visual Micro is a custom partition table. We add a new partitions_table.csv and follow the instructions.

You will also need to modify sdkconfig.h and update these defines:
#define CONFIG_PARTITION_TABLE_FILENAME "coredump_partitions.csv"
#define CONFIG_PARTITION_TABLE_CUSTOM_FILENAME "coredump_partitions.csv"

In this case I wanted to try a partition table with ota, a fat filesystem and a coredump.

# Name,   Type, SubType, Offset,   Size
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
eeprom,   data, 0x99,    0x290000,0x1000,
fat,      data, fat,     0x291000,0x15F000,
coredump, data, coredump,0x3F0000,0x10000

I've wrote a small snippet of code to display the partition table once the esp boots up and now we can upload the code to esp32:

platformio run -t upload

If we're successful, we'll see:

Leaving...
Hard resetting...

==== [SUCCESS] Took 20.79 seconds ====

Now we can monitor esp32:

platformio device monitor

And see the results:

- label           type/subtype   address:size            encrypted
- app0            0x00/0x10      0x00010000:0x00140000   0
- app1            0x00/0x11      0x00150000:0x00140000   0
- nvs             0x01/0x02      0x00009000:0x00005000   0
- otadata         0x01/0x00      0x0000e000:0x00002000   0
- eeprom          0x01/0x99      0x00290000:0x00001000   0
- fat             0x01/0x81      0x00291000:0x0015f000   0
- coredump        0x01/0x03      0x003f0000:0x00010000   0

Great! First success with the esp-idf and PlatformIO!

You can find the sources for this POC here:
https://github.com/drorgl/esp32-poc



Core Dumps

One of the features that an enterprise grade product needs is to be able to analyze crashes, not just the ones that a developer is able to attach a debugger to or the ones that can be reproduced but the ones that seldom happen, the ones that happen only in production, the hard ones.

To be able to do that, Alexey Gerenkov implemented saving the core dumps to a special flash partition, but I wanted to do something a bit different, I wanted to send the core dumps back to my server for monitoring, so whenever the device would start it will go to the core dump partition, check if it not empty, then be able to send it to the server, save it to the SPI fat file system or anything else that might be needed.

Documentation:
https://github.com/espressif/esp-idf/blob/master/docs/api-guides/core_dump.rst

I had a good starting point for a crash, I've used this code.

You need to select a partition table with a core dump partition, this is where the previous POC came into play as the PlatformIO esp-idf framework can only use partitions_singleapp.csv as far as I can tell without my modification.

I then started to look for a way to retrieve the core dump partition, you will need to use
esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_COREDUMP, NULL);

I've then used esp_partition_mmap to easily access the partition and check its length.

I think it might be better to check for the start and end markers instead of a "not empty" (0xFF) characte, but its good enough for the POC.
COREDUMP_FLASH_MAGIC_START
COREDUMP_FLASH_MAGIC_END

So now we have the start address and length of the core dump, we can do whatever we want with it, dump it to screen as base64, there's also a built in function that does that on crash, but you need to configure it:
#define CONFIG_ESP32_ENABLE_COREDUMP 1
#define CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE 0
#define CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH 1
#define CONFIG_ESP32_ENABLE_COREDUMP_TO_UART 1

Again, this is not what I wanted as there will be no one connected to the device to grab that core dump from the console.

Eventually we can assume we have a core dump, make sure you're uploading the core dump along with the firmware version it happened in otherwise you won't know where it crashed and you'll get the wrong results.

We can now run the command to get the core dump in text form:
espcoredump.py info_corefile -t b64 -c test_crash.txt -g toolchain-xtensa32\bin\xtensa-esp32-elf-gdb.exe ".pioenvs\esp32dev\firmware.elf" 

And the result:
===============================================================
==================== ESP32 CORE DUMP START ====================

================== CURRENT THREAD REGISTERS ===================
pc             0x400d0901       0x400d0901 <recur_func()+73>
lbeg           0x400014fd       1073747197
lend           0x4000150d       1073747213
lcount         0xffffffff       4294967295
sar            0x0      0
ps             0x60f20  397088
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x400d08e0       1074596064
a1             0x3ffc7f20       1073512224
a2             0x2      2
a3             0x3f403578       1061172600
a4             0x3ffc7f60       1073512288
a5             0x3ffc271c       1073489692
a6             0x0      0
a7             0x0      0
a8             0x5      5
a9             0xffffffad       -83
a10            0x20     32
a11            0x3ffc656c       1073505644
a12            0x1      1
a13            0x80     128
a14            0x1      1
a15            0x0      0

==================== CURRENT THREAD STACK =====================
#0  0x400d0901 in recur_func () at src\ est_core_dump.cpp:80
#1  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#2  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#3  0x400d0924 in unaligned_ptr_task (pvParameter=0x0) at src\  est_core_dump.cpp:90

======================== THREADS INFO =========================
  Id   Target Id         Frame
  9    process 8         xQueueGenericReceive (xQueue=0x3ffc2ecc, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  8    process 7         xQueueGenericReceive (xQueue=0x3ffc2904, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  7    process 6         prvTimerTask (pvParameters=0x0) at .platformio\\packages\\framework-espidf\\components\\freertos\     imers.c:445
  6    process 5         failed_assert_task (pvParameter=0x0) at src\   est_core_dump.cpp:103
  5    process 4         0x400d0c12 in init_analysis_task (pvParameter=<optimized out>) at src\ est_core_dump.cpp:130
  4    process 3         bad_ptr_task (pvParameter=0x0) at src\ est_core_dump.cpp:51
  3    process 2         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
  2    process 1         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
* 1    <main task>       0x400d0901 in recur_func () at src\    est_core_dump.cpp:80

======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x400c0000 0x0 RW
.iram0.vectors 0x40080000 0x400 R XA
.iram0.text 0x40080400 0x7860 R XA
.dram0.data 0x3ffc0000 0x1d20 RW A
.flash.rodata 0x3f400010 0x3e64 RW A
.flash.text 0x400d0018 0xed20 R XA
.coredump.tasks 0x3ffc6504 0x164 RW
.coredump.tasks 0x3ffc7e60 0x1d4 RW
.coredump.tasks 0x3ffc559c 0x164 RW
.coredump.tasks 0x3ffc5430 0x160 RW
.coredump.tasks 0x3ffc5028 0x164 RW
.coredump.tasks 0x3ffc4eb0 0x16c RW
.coredump.tasks 0x3ffc6398 0x164 RW
.coredump.tasks 0x3ffc76e0 0x14c RW
.coredump.tasks 0x3ffc622c 0x164 RW
.coredump.tasks 0x3ffc6ed0 0x154 RW
.coredump.tasks 0x3ffc8848 0x164 RW
.coredump.tasks 0x3ffc86f0 0x14c RW
.coredump.tasks 0x3ffc6004 0x164 RW
.coredump.tasks 0x3ffc5ea0 0x158 RW
.coredump.tasks 0x3ffc2d60 0x164 RW
.coredump.tasks 0x3ffc2bd0 0x184 RW
.coredump.tasks 0x3ffc3328 0x164 RW
.coredump.tasks 0x3ffc3190 0x18c RW

===================== ESP32 CORE DUMP END =====================
===============================================================


CPU Utilization

This is a bit of a hack, since I would rather not change esp-idf code without commiting the change back to the main repo, I've tried many workarounds, eventually the last nail in the coffin was the fact there's a definition for portGET_RUN_TIME_COUNTER_VALUE in freertos/portmacro.h, making it impossible to override from outside the framework.

So.... I had to modify portmacro.h, a change I plan to ask to push back into esp-idf:

#ifndef portGET_RUN_TIME_COUNTER_VALUE
#define portGET_RUN_TIME_COUNTER_VALUE()  xthal_get_ccount()

#endif

Then I needed to add the functions for initializing the counter and getting the counter values in sdkconfig.h:

#ifndef __ASSEMBLER__
#ifdef __cplusplus
extern "C"{
#endif

#ifdef ESP32
extern unsigned long get_run_time_counter_value();
extern void init_run_time_counter();
#endif

#ifdef __cplusplus
}
#endif

#define portGET_RUN_TIME_COUNTER_VALUE() get_run_time_counter_value()
#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() init_run_time_counter()
#endif

#define configGENERATE_RUN_TIME_STATS 1
#define configUSE_STATS_FORMATTING_FUNCTIONS 1

#define configUSE_TRACE_FACILITY 1

(In case you're wondering about the #ifndef __ASSEMBLER__ its because the sdkconfig.h is included in .S files)

I then went ahead and used system_get_time to get the value for the counter as it was the only stable one for now, the others are core dependent so every time a task would switch core due to context switching, the counter value will be different showing skewed results.

I know system_get_time is deprecated but I've yet to find a better function with stable results that won't crash, I've tried gettimeofday but it crashed the device.

In any case, this should not be used in production software according to FreeRTOS.

After running for a few cycles, we get these results:

Tasks currently running:
Printing Stats:
high_load1      502167712        49
medium_load2    15551453         1
display_stats   235900          <1
medium_load1    18987019         1
low_load2       384767          <1
low_load1       422990          <1
IDLE            34476           <1
IDLE            977132          <1
high_load2      505107417        50
Tmr Svc         60              <1
ipc0            340010          <1
ipc1            22441           <1

iterations low: 70 medium: 3156 high: 100096

What I would like you to notice is the amount of iterations each function executed, bear in mind this is an RTOS, so higher priority tasks run first, if the high and medium tasks are not in a wait section such as vTaskDelay, the low priority won't get any time, taskYIELD might not be enough here so make sure you don't have high priority tasks that never release CPU time.


Summary

This was a short demonstration of a few things the ESP32 can do in terms of enterprise grade devices, the more I read esp-idf code, the more I realize its potential. 

I think this development board is one of the most fun ones I played with, in the upcoming weeks I'll see if its possible to apply enterprise grade architecture and design with it, I've already implemented a lean dependency injection for it and did a POC controlling it with Azure IoT Hub. 

You must have seen that:




In my opinion ESP32 + FreeRTOS + Azure IoT Hub + PlatformIO is overlooked and has the potential of realizing many Industry 4.0 projects.

To understand the potential, this is a small section from Microsoft Build 2017 First day Keynotes, you will notice what analyzing seemingly naive data can give you in the long run a change in power consumption, vibrations, temperatures can all point to an imminent failure, predicting maintenance, short product life and much much more.



I think the next best step will be to learn about PlatformIO unit testing capabilities.

I'm looking forward to see what espressif will think of next, just imagine the PC we had about 20 years ago is now the size of a coin and can be programmed by anyone.

Tags: , , ,