RISC-V Linux on ESP32

I've been playing with the idea of running linux on ESP32 since the first days I've met its more robust module, the WROVER-B, on paper it seem possible since its a dual core 240Mhz and has 16MB flash and 8MB RAM, compared to our antique machines that could run linux, it seems like a beast.

Doing some research on it, I've understood that its MMU is insufficient for running Linux on it. during the past few years I've been looking into it to see if anyone else found the time to implement it and eventually I've decided its going to be a good opportunity to learn a bit more about RISCV and Buildroot. Two subjects I've been putting off for longer than I'd like to admit.

I've decided to start with something rather to write it all from scratch, which I didn't have time or energy to do for this project, I've looked into QEMU emulation for RISCV but taking this project apart and getting only a few components out of it to run on an embedded system seemed like too much work. Eventually I've found out about Fabrice Bellard's TinyEMU (demo).

RISC-V

RISC-V is the new ISA kid in the block, well, not really a kid and not really new, but it becomes more and more popular, Espressif got out the ESP32-C3 at 2020.

Previous Successes

Max Filippov patched the kernel to support ESP32 back at 2019, I'm pretty sure it runs a lot faster since its not an emulation.

Li XiongHui wrote the juiceVM which implemented RISCV ISA and runs on ESP32, he wrote about it on whycan and reddit and has video of it booting on YouTube, the video does state x30 speedup, which means the system booted in about 6 hours. Li never released the source code so the only improvements that can be done is by him and judging from my own life, you never have enough time for these things.

TinyEMU

"TinyEMU is a system emulator for the RISC-V and x86 architectures. Its purpose is to be small and simple while being complete."

Looking at its source code, seemed like the project went from mission impossible (with my current resources) to mission possible. Oh the naiveté.

ESP32

The ESP32 is a dual core 240Mhz MCU, it was released at September 2016 and its still one of the best value for money MCU you can get, one of its versions has 8MB or RAM and 16MB of FLASH. That amount of RAM on any of its competitors takes more than "I want it" to get it working, at the time it came out especially so. Espressif did an amazing job with esp-idf and one of their best features is listening to their customers and with the help of the maker community they've built an amazing framework.

TinyEMU on ESP32

Will it even compile?

Apparently yes, making it compile was very easy, some tweaks here and there and a missing standard library and it was compiled perfectly. 

But what can I do about memory? the ESP32 only has 8MB and half of it is not even accessible as a standard but rather bank switched with its own APIs (himem).

My thinking at the time was that it doesn't matter so much since the kernel will probably not need so much memory once it starts, for example, if I don't access files, the memory holding the file system and the file system functions will be left alone other than a periodic flush.

So something like a swap file will probably be good enough for this experiment, right?

Not so fast.

First, I had to find out that the standard way of accessing the SD card is not fast enough at around 150k per second. So I went to a journey to find all the bottlenecks.

Then I've discovered that 3MB for the virtual pages is too slow due to the PSRAM, using 80Mhz only improved a bit, so I went on a journey to find the fastest search trees, I've tried splay tree and eventually rested on AVL tree which was good enough.

But I could squeeze more out of the ESP32, I've implemented a rudimentary direct-mapped-cache so most memory accesses won't even search for their page (Professor Luis Ceze has a great presentation on the subject).

I was still not happy enough, my SD card is slow and no matter what I did, it slowed things down. I've measured how many page faults I had and decided that dirty pages should only be written once they are abandoned, so my LRU cache pushed dirty pages into himem and only when these himem pages reclaimed they got pushed to the page file.

At that point I was content enough, my kernel booted in 1:35 minutes. 

UPDATE 2023-04-13: Improved Speed by 40%

Buildroot

"Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation."

I've always wanted to learn how the big guys do it, how embedded cameras, kiosks and perhaps even satellites gets their OS build without all the bloat and package managers.

So once I got my basic emulator working (without all the optimizations), it was time to start learning buildroot. 

Apparently its simple, you just download the archive, work through a few menus, read some documentation, modify the rootfs with overlays and you're done. issue a make command and you have your kernel and your rootfs.

In the old days (or so I've heard), you've had to build RISC-V toolchain and patch the kernel to get things going, These days buildroot comes with a precompiled toolchain from bootlin, so the whole experience was fun and easy to learn.

Thoughts for the future

Using the emulator to run embedded RISC-V code, implement SiFive GPIO and provide more flexible VM for running code on multiple embedded platforms. Communicating over the console is implemented in TinyEMU patches, or see here or through virtio.

Closing

This was a wonderful journey of learning, I've learned more than I wanted about:

• file systems (thank you fatfs)
• buildroot
• himem
• RISC-V Device Tree and FDT (Flattened Device Tree)
• Proxy Kernel and BBL (Berkeley Boot Loader)
• SBI (Supervisor Binary Interface)
• hvc0
• virtio
• memory layouts and caches
• kernel memory and size optimizations (here, here and here).

You can find the fruits of this labor at:

• ESP32 VMM TinyEMU Port https://github.com/drorgl/esp32-tinyemu
• Buildroot for RISC-V 32 Getting Started https://github.com/drorgl/buildroot-tinyemu

Tags: buildroot, esp32, linux, riscv, riscv32, riscv64, tinyemu