Automount SD cards #10129

tannewt · 2025-03-12T23:29:39Z

And make them available over USB MSC. They can be remount to make them writable (slowly) from the host PC.

And make them available over USB MSC. They can be remount to make them writable (slowly) from the host PC. Fixes micropython#9954. Fixes micropython#8678. Fixes micropython#3477

RetiredWizard · 2025-03-22T20:02:42Z

I have been able to cause safemode crashes accessing the automounted SD cards. I've been trying to reduce the code needed to reproduce and this is what I currently have as code.py:

import sdioio
import board
import storage,os

sd = sdioio.SDCard(clock=board.GP36,command=board.GP35,data=[board.GP37, board.GP33, board.GP38, board.GP34],frequency=25000000)
vfs=storage.VfsFat(sd)
storage.mount(vfs,'/sd')
os.chdir('/sd')

while True:
    tFSize = 0
    nFiles = 0
    nDirs = 2
    print(f'Directory of {os.getcwd()}.')
    for dir in os.listdir():
        if os.stat(dir)[0] & (2**15)== 0:
            print(f'{dir}{" "*(24-len(dir))}<DIR>{" "*18}')
            nDirs += 1

    tmpDir = os.getcwd()
    try:
        availDisk = os.statvfs(tmpDir)[1]*os.statvfs(tmpDir)[4]
    except:
        availDisk = 0
    for dir in os.listdir():
        if os.stat(dir)[0] & (2**15) != 0:
            fSize = str(os.stat(dir)[6])
            tFSize += os.stat(dir)[6]
            print(f'{dir}{" "*(35-len(dir)+10-len(fSize))} {fSize}')
            nFiles += 1

My current theory/guess is that this has something to do with accessing the SD card during the process of mounting it to the host computer. I put a 20 second delay between the mount and the start of the directory listing and that seems to prevent the crash. My host PC is running Ubuntu.

It doesn't always result in a crash but putting ~100 files and a couple empty sub directories on the SD card seems to make it crash more reliably. I originally noticed the crash using an SD card with a linux boot image so there were actually two partitions automounted and more than 100 files being listed. I have also formatted a 32G SD card and seen the error on that as well.

Before the board hard crashes I sometimes get the following error:

Traceback (most recent call last):
  File "code.py", line 26, in <module>
OSError: [Errno 2] No such file/directory

I'm using the Waveshare S3 Geek board which has a display and since my current theory is this is a timing/race condition issue outputting to the LCD screen may be part of the issue. I don't think I have a board that has an SD card but no display, but I'll look around.

RetiredWizard · 2025-03-22T22:25:21Z

I assume this isn't actually supposed to automatically mount the SD card to CircuitPython but when an SD card is mounted it "automounts" the cards to the host computer. That's what I'm seeing anyway.

On the sparkfun thingplus RP2040 (no display) the automount to the Host PC does occur, but it takes considerably longer than the S3 board. I was also unable to get the code.py (slightly altered to use sdcardio and busio) to cause the crash, although with the code.py running, the mount to the host PC didn't occur before I gave up (a couple minutes).

tannewt · 2025-03-27T18:33:45Z

Does the Waveshare S3 Geek share the SPI bus between the display and SD card?

RetiredWizard · 2025-03-27T18:40:45Z

Does the Waveshare S3 Geek share the SPI bus between the display and SD card?

It looks like two separate SPI busses...

dhalbert

I tried the artifact build on a PyPortal with a 4GB SD card in the slot.

With no code.py I can see some attempt to mount the SD card, but it does not complete. From /var/log/syslog:

2025-03-28T11:00:38.921872-04:00 cod kernel: sd 4:0:0:1: [sdc] 7744512 512-byte logical blocks: (3.97 GB/3.69 GiB)
2025-03-28T11:00:38.923918-04:00 cod kernel: sdc: detected capacity change from 0 to 7744512

But it never shows up in mount or df. This is on Ubuntu 24.04. sdb1 is CIRCUITPY.

If I try running the NASA PyPortal demo, CircuitPython hangs and boot loops (slowly, many seconds between tries)

dhalbert · 2025-03-28T15:40:02Z

Some more testing. The detected capacity change shown above is not typical. This is from a cold start, no code.py, PyPortal, with either a 4GB or a 64MB card in the SD slot:

2025-03-28T11:36:40.581905-04:00 cod kernel: scsi 4:0:0:0: Direct-Access     Adafruit PyPortal         1.0  PQ: 0 ANSI: 2
2025-03-28T11:36:40.581939-04:00 cod kernel: scsi 4:0:0:1: Direct-Access     Adafruit PyPortal         1.0  PQ: 0 ANSI: 2
2025-03-28T11:36:40.582861-04:00 cod kernel: sd 4:0:0:0: Attached scsi generic sg2 type 0
2025-03-28T11:36:40.583893-04:00 cod kernel: sd 4:0:0:1: Attached scsi generic sg3 type 0
2025-03-28T11:36:40.584884-04:00 cod kernel: sd 4:0:0:0: [sdb] 16377 512-byte logical blocks: (8.39 MB/8.00 MiB)
2025-03-28T11:36:40.585861-04:00 cod kernel: sd 4:0:0:0: [sdb] Write Protect is off
2025-03-28T11:36:40.585883-04:00 cod kernel: sd 4:0:0:0: [sdb] Mode Sense: 03 00 00 00
2025-03-28T11:36:40.585889-04:00 cod kernel: sd 4:0:0:1: [sdc] Media removed, stopped polling
2025-03-28T11:36:40.586819-04:00 cod kernel: sd 4:0:0:0: [sdb] No Caching mode page found
2025-03-28T11:36:40.586824-04:00 cod kernel: sd 4:0:0:0: [sdb] Assuming drive cache: write through
2025-03-28T11:36:40.586825-04:00 cod kernel: sd 4:0:0:1: [sdc] Attached SCSI removable disk
2025-03-28T11:36:40.605825-04:00 cod kernel:  sdb: sdb1
2025-03-28T11:36:40.605834-04:00 cod kernel: sd 4:0:0:0: [sdb] Attached SCSI removable disk
2025-03-28T11:36:40.611107-04:00 cod (udev-worker)[156601]: sdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc' failed with exit code 1.
2025-03-28T11:36:40.740286-04:00 cod (udev-worker)[156628]: sdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb' failed with exit code 1.
2025-03-28T11:36:40.848223-04:00 cod (udev-worker)[156621]: sdb1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb1' failed with exit code 1.
2025-03-28T11:36:40.991859-04:00 cod kernel: audit: type=1107 audit(1743176200.990:1207): pid=1409 uid=101 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/login1" interface="org.freedesktop.DBus.Properties" member="PropertiesChanged" name=":1.13" mask="receive" pid=5288 label="snap.thunderbird.thunderbird" peer_pid=1440 peer_label="unconfined"
2025-03-28T11:36:40.991867-04:00 cod kernel:  exe="/usr/bin/dbus-daemon" sauid=101 hostname=? addr=? terminal=?'
2025-03-28T11:36:41.016837-04:00 cod kernel: FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
2025-03-28T11:36:41.017110-04:00 cod udisksd[1442]: Mounted /dev/sdb1 at /media/halbert/CIRCUITPY on behalf of uid 1000
2025-03-28T11:36:41.017821-04:00 cod kernel: audit: type=1107 audit(1743176201.016:1208): pid=1409 uid=101 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/login1" interface="org.freedesktop.DBus.Properties" member="PropertiesChanged" name=":1.13" mask="receive" pid=5288 label="snap.thunderbird.thunderbird" peer_pid=1440 peer_label="unconfined"
2025-03-28T11:36:41.017826-04:00 cod kernel:  exe="/usr/bin/dbus-daemon" sauid=101 hostname=? addr=? terminal=?'

sdc shows up, but never gets mounted.

Also tested on Windows 11: Usually, neither CIRCUITPY nor the SD card show up. Once, both showed up as D: and E:, but were non-functional.

On macOS 15.3.2: CIRCUITPY shows up, but there's no sign of the other drive in System Information.

tannewt · 2025-03-28T16:59:43Z

Thanks for the testing! I'll get back to this next week.

tannewt · 2025-03-31T20:37:14Z

@RetiredWizard I think your crash may be due to sdioio. I don't think it is designed for native use. I'm adding checks to prevent it from showing over USB MSC.

I'm testing with automounted sdcardio which is allocated outside the VM and has a native API to use from USB MSC.

Add Metro RP2350 definitions

tannewt · 2025-03-31T22:28:41Z

Ok, I've made the MSC LUN -> filesystem checks stricter. Only sdcardio that is not allocated to the VM heap will be exposed my USB MSC. This basically requires the automounting definitions for the board. I've added them to Metro RP2350 in addition to Fruit Jam. Other ports/boards will be need to be added as they can be tested.

I tested on the PyPortal briefly but had issues with sdcardio that I didn't want to debug. The same card works on RP2350. PyPortal shares the clock line between the SD card and the ESP32.

I also tested on Mac in addition to Linux and the SD card appears and disappears as I expect on both platforms.

tannewt · 2025-03-31T22:31:05Z

The mac was my old 2019 Macbook Pro running 15.0.1.

dhalbert

Tested with a Metro RP2350 on Ubuntu 24.04, Windows 11, and macOS 15.3.2.
All work!

The SD card is normally read-only to the host computer.
If I put import storage; storage.remount("/sd", readonly=True) in boot.py, it becomes R/W to the host. Doing that after USB enumeration, say in the REPL, does not change the host view of the drive, but does not give an error.

Maybe raise an error if storage.remount() is attempted after USB starts?
Document the feature, maybe after this section on CIRCUITPY? https://docs.circuitpython.org/en/latest/docs/workflows.html#circuitpy-drive

tannewt · 2025-04-01T18:24:52Z

I've added more documentation. I think it may have worked for you but you didn't realize. It takes a few seconds for CP to "eject" and then make the drive available again.

anecdata · 2025-04-01T18:30:25Z

In the case of CIRCUITPY, storage.remount() can only be done in safemode.py or boot.py. Is that also true for CPSAVES and SD?

tannewt · 2025-04-01T18:35:41Z

In the case of CIRCUITPY, storage.remount() can only be done in safemode.py or boot.py. Is that also true for CPSAVES and SD?

It is not strictly true for CIRCUITPY. If you can eject CIRCUITPY from your host OS (like Linux can), then you can remount it. What happens is that the USB writable check grabs a lock on the underlying block device. On Linux, this happens before you actually "mount" the drive. It is only released on eject. boot.py runs before this so it always works.

CPSAVES and SD don't have this same problem because they are read-only to the host OS by default. Therefore, the initial USB discovery doesn't grab the lock. Once they are remounted read-write to the host, then the lock is grabbed and only released on eject.

CircuitPython only grabs the underlying write lock when a file is open for writing. It knows more about when manipulations are happening.

dhalbert

Thanks for the documentation!

tannewt added 2 commits March 12, 2025 16:27

Automount SD cards

d0c8400

And make them available over USB MSC. They can be remount to make them writable (slowly) from the host PC. Fixes micropython#9954. Fixes micropython#8678. Fixes micropython#3477

Fix boards without USB MSC

0801f3c

tannewt marked this pull request as ready for review March 13, 2025 18:55

tannewt requested a review from dhalbert March 14, 2025 20:34

Neradoc mentioned this pull request Mar 22, 2025

Problems with accessing SD CARD on ESP32-S3-GEEK board #10161

Closed

dhalbert requested changes Mar 28, 2025

View reviewed changes

Merge remote-tracking branch 'adafruit/main' into automount_sd

2c0f9e9

Be stricter about what filesystems it works with.

cb97ae6

Add Metro RP2350 definitions

dhalbert requested changes Mar 31, 2025

View reviewed changes

tannewt added 2 commits April 1, 2025 10:30

Merge remote-tracking branch 'adafruit/main' into automount_sd

e8875f5

Add additional documentation

2e41242

tannewt requested a review from dhalbert April 1, 2025 22:49

dhalbert approved these changes Apr 2, 2025

View reviewed changes

dhalbert merged commit 1d51d95 into adafruit:main Apr 2, 2025
612 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automount SD cards #10129

Automount SD cards #10129

tannewt commented Mar 12, 2025

RetiredWizard commented Mar 22, 2025 •

edited

Loading

RetiredWizard commented Mar 22, 2025

tannewt commented Mar 27, 2025

RetiredWizard commented Mar 27, 2025

dhalbert left a comment

dhalbert commented Mar 28, 2025

tannewt commented Mar 28, 2025

tannewt commented Mar 31, 2025

tannewt commented Mar 31, 2025

tannewt commented Mar 31, 2025

dhalbert left a comment

tannewt commented Apr 1, 2025

anecdata commented Apr 1, 2025

tannewt commented Apr 1, 2025

dhalbert left a comment

Automount SD cards #10129

Automount SD cards #10129

Conversation

tannewt commented Mar 12, 2025

RetiredWizard commented Mar 22, 2025 • edited Loading

RetiredWizard commented Mar 22, 2025

tannewt commented Mar 27, 2025

RetiredWizard commented Mar 27, 2025

dhalbert left a comment

Choose a reason for hiding this comment

dhalbert commented Mar 28, 2025

tannewt commented Mar 28, 2025

tannewt commented Mar 31, 2025

tannewt commented Mar 31, 2025

tannewt commented Mar 31, 2025

dhalbert left a comment

Choose a reason for hiding this comment

tannewt commented Apr 1, 2025

anecdata commented Apr 1, 2025

tannewt commented Apr 1, 2025

dhalbert left a comment

Choose a reason for hiding this comment

RetiredWizard commented Mar 22, 2025 •

edited

Loading