• Skip to main content
  • Skip to primary sidebar

RNTLab.com

The Ultimate Shortcut to Learn Electronics and Programming with Open Source Hardware and Software

  • Courses
  • Forum
    • Forum
    • Ask Question
  • Shop
  • Account
  • Blog
  • Login

Efficient way to log fast data on mircoSD

Q&A Forum › Category: ESP32 › Efficient way to log fast data on mircoSD
0 Vote Up Vote Down
Christopher Stillwell asked 4 years ago

I am currently trying to log the data of a IMU (LSM9DS1) and save the data locally to be later processed in MATLAB/Python. The main issue that I have run into is the low number of samples saved to the micro SD card. When saving all the values from the accelerometer, gyroscope, and magnetometer I have around 14-15 data points per second (goal is about 60 per second). At first I thought it was an issue with the I2C bus being too slow and I set the clock to 400kHz, but the sample rate stayed the same.
I did some rudimentary tests using only the gyro at 952Hz and the I2C at 100kHz using the code with some added serial print statements and millis():

https://pastebin.com/LsHjjQs5

I got the sample rate it takes to only check if the gyro is available is about 917Hz. The sample rate it takes to check if the gyro is available and read the values is 549Hz (902Hz when I2C is at 400kHz). The sample rate to convert the saved values into a string is 540Hz (898Hz when I2C is at 400kHz). When finally appending the values to a file, the sample rate drops down to an average of 30-46Hz no matter what the I2C speed is set to.
I’m assuming most of the time is spent opening the file instead of writing to it. I am using the appendFile function that was used in “Learn ESP32 with Arduino IDE” as seen below:

https://pastebin.com/dXuJ9iH6

I read somewhere else about creating a buffer and waiting for it to fill up before writing to the file. I was wondering if anyone else has done something similar, or has potential solutions to increase the saved sample rate, or just point me in the right direction. Thanks!
The hardware I have is an ESP32 Huzzah, SanDisk Ultra microSD 32GB, and a sparkfun LSM9DS1 (the stick).

Question Tags: IMU Data Logging on ESP32
4 Answers
0 Vote Up Vote Down
Sara Santos Staff answered 4 years ago

Hi Christopher.
What’s the microSD card module that you’re using?
With the ESP32, you can communicate with the microSD using SPI communication or using the SDMMC controller.
I think you can write data faster using the SDMMC controller.
This article explains in great detail how to maximize the speed: https://www.reddit.com/r/esp32/comments/d71es9/a_breakdown_of_my_experience_trying_to_talk_to_an/.
I hope this helps.
Regards,
Sara
 

0 Vote Up Vote Down
Christopher Stillwell answered 4 years ago

Thanks for the quick response. I currently have a SparkFun MicroSD Transflash Breakout. I will try the 1 bit mode for SDMMC and see how that goes. Thanks again for the response.

0 Vote Up Vote Down
Sara Santos Staff answered 4 years ago

Hi.
That microSD card breakout board is compatible with SDMMC. So, you can try it 🙂
Then, let me know how it went.
Regards,
Sara

0 Vote Up Vote Down
Barry answered 4 years ago

A few points:

  1. How much data (how many bytes) in a sample of the IMU. Use bytes for the calculations as transactions are in bytes. Multiply sample size x samples/second. 
  2. Now calc the I2C bit rate needed to handle about 2x the data size.
  3. It takes a lot of time(cpu cycles) to open/close a file. Make a buffer to store about 25% of the data, then write it. 

You’ll need to balance the sample rate & buffer size to see what works.
Consider repeated writing of a test pattern to see how fast you can write to the file. It might not be very fast.
 

Primary Sidebar

Login to Ask or Answer Questions

This Forum is private and it’s only available for members enrolled in our Courses.

Login »

Latest Course Updates

  • [eBook Updated] Learn Raspberry Pi Pico/Pico W with MicroPython eBook – Version 1.2 May 26, 2025
  • [New Edition] Build ESP32-CAM Projects eBook – 2nd Edition April 16, 2025

You must be logged in to view this content.

Contact Support - Refunds - Privacy - Terms - MakerAdvisor.com - Member Login

Copyright © 2013-2025 · RandomNerdTutorials.com · All Rights Reserved

Insert/edit link

Enter the destination URL

Or link to existing content

    No search term specified. Showing recent items. Search or use up and down arrow keys to select an item.