ESP32 SD Card Optimization

 Reading and writing SD Cards with ESP32 should be simple, however, the amount of moving parts in esp-idf makes that a complicated task, first to understand and then to optimize.

Lets disect the stack:

File Operations

Working with files should not be new to you, however, there are some differences between ESP32 and working on a full pledged OS, these differences can affect how the product will behave.

On OS the buffered and unbuffered behaves very similar, in the end its the OS's job to buffer the disk operations so the system and applications are more responsive

However, due to resource limitations, on an MCU there is no such buffering other than the Standard library buffering and disk operations requirements.

In turn, this means you'll need to use the API in a manner suitable for these limitations.

Its probably best to avoid the buffered APIs if you don't plan to write unstructured data, such as logs and textual information, the unbuffered calls will pass the read and write requests as-is, so you can have as much control as you need without modifying any of the libraries.

Standard Library

The standard library is responsible for translating the buffered and unbuffered calls to the next layer, so eventually these functions will call the platform's implementation. 

Specifically in ESP32, newlib was compiled with 128 bytes buffer, this creates very inefficient calls to the file system since each fread/fwrite will split each call to 128 bytes which is significantly less than the sector size.

You may ease some of this inefficiency by increasing the buffer size for the buffered calls:

if(setvbuf(file, NULL, _IOFBF, 4096) != 0) {
  //handle error
}

File Stream Adapter

The file stream adapters, or the VFS enables mounting multiple file systems under different root paths, so in essence each root path refers to a different mount.

When opening a file through the standard library, a new file descriptor is created in the file system driver (currently only FATFS), it then gets translated into VFS file descriptor and returned to the standard library.

When writing to the file descriptor, the process reverses and the file descriptor is translated back to the filesystem's file descriptor.

File System

Currently only FATFS is implemented as file system (FAT) in ESP32, however, using other file systems such as LittleFS requires very little effort.

The file system is responsible for opening, reading, writing and other file system operations to read and persist from a physical medium.

For example, when opening a file, the FATFS will go to the root directory and traverse subdirectories until it finds where the file is located while reading the file allocation table, all of that is done with raw access to the underlying storage, i.e. read sector x,y,z.

FATFS was designed for resource constrained systems, so it will not buffer more than it must and will read and write as little as possible to achieve the task it needs, even if it means inefficient calls to the storage. for example, writing unaligned data will result in read sector, update part of it, write it back.

This creates inefficiencies when working with unaligned random accesses to files since it requires FATFS to "waste" time getting the data it needs in blocks and not using part of that data.

But these inefficiencies can be reduced by making sure all your random access to files is aligned to the sector sizes. 

More issues you may encounter is when your reads and writes cross the cluster boundaries, each cluster can be hosted on a different place in the block device which will require two separate calls to the underlying storage.

To reduce the amount of waste these calls create, FATFS implements fast seek (and in esp-idf), which caches the cluster link map table, however this works only on read only files or preallocated write files.

Also, if you plan to use your design as a product, keep in mind that Windows formats SD cards as EXFAT which is turned off by default in esp-idf. this can cause some usability issues to your customers and if you choose you can override that configuration. There are two ways of doing it, each with its own advantages and drawbacks:

1. copy the component/fatfs to your project's root directory under component/fatfs and rebuild the project.

2. Override package files and enable it in your ffconf.h.

File System Adapter

The file system adapter's job is to read and write data from a block device, optionally with wear leveling driver in between, since SD card implement wear levelling internally, this layer is not used when working with SD cards.

Most block devices such as SD cards and eMMC implement their storage in blocks, usually 512 or 1024 bytes per block (or sector), which means that the FATFS layer must speak in sectors, so if it needs to update 2048 bytes, it will do it in 512 bytes chunks in most cases.

Hardware Drivers

The hardware driver's job is to read and write sectors to the underlying storage media, currently SD SPI and SDMMC 1 and 4 bits are impended.

Recommendations:

• Configure your hardware drivers as fast as possible, SPI and SDMMC. 
• Prefer Sequential over Random reads and writes, SD cards were designed for sequential access.
• If you must use random access, avoid using the buffering APIs 
• If you must use random read / write, prefer aligned read and write in sector size chunks.
• If you must use buffering APIs, increase the buffer to at least the sector size in sector size increments (your average buffer size aligned to 512 or 1024 )
• Use FATFS cluster cache (or fast seek) if applicable, please note that to reduce incompatibilities the cache is disabled for writeable files in vfs_fat.c, however, you can override it if you know what you're doing.
• Enable EXFAT if its a customer facing product
• Prefer SDMMC over SPI, though it requires pull ups and you might need to adjust your design due to GPIO bootloader modes, however, I was able to use high speed sdmmc by activating the internal pullup on GPIO 12 programmatically and it was enough.

This has been a learning experience, hopefully my lessons will ease your learning experience.

Relevant github repo: https://github.com/drorgl/esp32-sd-card-benchmarks

Tags: esp-idf, esp32, exfat, fat, fatfs, fopen, fread, fseek, fwrite, sd, sdmmc