Investigate an Ipfs Issue Leads to Finding Out Corrupted Sd Card

Symptom

ipfs get bafybei… (a ~333 MiB DAG) on the requesting host hangs at exactly 113.16 MiB / 333.12 MiB (33.97%). The CID is “known to be on the private network” (4 RPi5 IPFS nodes, all installed from the same script).

Investigation

1. Bitswap state on the requesting node

ipfs bitswap stat

bitswap status
    blocks received: 4
    blocks sent: 0
    wantlist [1 keys]
        bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku
    partners [0]

Hung waiting on a single raw block. Despite 4 swarm peers being connected (four hosts on the LAN, last-octet labels .1/.2/.3/.4), bitswap reports partners [0].

2. Per-peer bitswap ledgers

3. node-1 (.1) logs

clk nd logs --job ipfs --node node-1 --stderr shows a Go runtime panic on every startup, in quic-go, on arm64:

exec ipfs daemon --migrate=true --enable-pubsub-experiment
INFO failed to sufficiently increase receive buffer size ...
runtime: pcdata is 7 and 5 locals stack map entries for
  github.com/quic-go/quic-go.(*outgoingStreamsMap[go.shape.*uint8]).OpenStream
fatal error: bad symbol table

Repeated across many supervisor restarts. The panic is in runtime.(*stkframe).getStackMap during copystack — classic symptom of an invalid pclntab / corrupted binary or memory.

4. Hash the binary inside the image

Different hashes for the same image tag. docker rmi --force + re-pull on node-1 produced fc63... again. So the image fetched into node-1’s docker storage is deterministically different from the one on node-2. The same tag in the registry is the same; docker image inspect would have settled which case applies, but the actual root cause was revealed by the next step.

5. Read-stability test on the binary

Storage layout on node-1:

• root FS on SD card (/dev/mmcblk0p2, 14 GB)
• Docker storage driver: overlay2 on extfs
• Last ext4 fsck: Sat Aug 17 12:11:40 2024, mount count 84, state “clean”, errors-behavior “Continue”
• dmesg/journal: no I/O errors (box was just rebooted today, ring buffer was empty)

Read the same on-disk file twice with the page cache flushed in between:

F=/var/lib/docker/overlay2/5f4262.../diff/usr/local/bin/ipfs
sha256sum "$F"                       # -> fc63cb93...
echo 3 > /proc/sys/vm/drop_caches
sha256sum "$F"                       # -> 931a207d...

Two different hashes for the same file in the same fs, no writes in between. The block device is returning different bytes on successive reads.

Root cause

The SD card on node-1 is silently corrupting reads (or has corrupted writes that now read back as random garbage). No kernel error, no ext4 error — just bit-flipping NAND. ext4 metadata_csum catches metadata corruption but file data has no checksum. Classic end-of-life SD card after a long stretch of Docker-style write load.

This explains the whole chain:

• Go runtime panics with “bad symbol table” → the executable’s pclntab / itab data read back as random bytes.
• docker rmi --force + re-pull doesn’t fix it → new bytes land on the same flaky NAND blocks (or the corruption is on read, not write).
• Hash differs from node-2 → node-2’s SD reads its bytes correctly.
• Same wrong hash twice in a row earlier in the session → page cache; flushing it surfaces the instability.
• Why the network used to serve this CID and now doesn’t → node-1 is the (only?) node that has the leaf block; its daemon can no longer stay up to serve it.

Remediation

1. Replace node-1’s SD card. Every read can return different garbage; nothing on it is trustworthy.
2. Before replacing, attempt to rsync the ipfs blocks volume off node-1; corrupted blocks will be rejected by ipfs on the receiving side (CID mismatch) so we keep whatever survives.
3. Verify on .2/.3/.4 whether any other node has the missing leaf block:

   clk nd exec ipfs <node> -- ipfs block stat --offline \
     bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku
   

   If yes, the current ipfs get can finish once node-1 stops poisoning the bitswap session, or with a one-shot ipfs swarm connect refresh after disabling node-1’s job.
4. When replacing, pick a quality card — see Raspberry Pi SD Card (SanDisk A1 / Extreme Pro hold up under Pi workloads). USB-SSD or NVMe HAT remain the safest option for heavy write load.

Can the corrupted card be reused after reformatting?

A destructive write+read test (f3probe, badblocks) can prove the card dead — any reported corruption is definitive. It cannot prove the card alive: consumer SD exposes no wear telemetry, and a passing run only certifies the cells it happened to touch, at that instant, not their retention weeks later. With bit-flips already observed there is no verdict left to chase — bin it.

Periodic detection — nomad sysbatch sentinel

The cheapest reliable detector for the failure mode we hit is the same one that found it: read a static file twice with the page cache flushed between reads, alert on hash divergence. Add a 1 MiB write+read-back to also catch write-side rot.

job "sdrot" {
  datacenters = ["dc1"]
  type = "sysbatch"

  # Read-stability sentinel for the root filesystem. Detects silent
  # NAND bit-flips that ext4 cannot — same technique that caught the
  # node-1 failure.

  periodic {
    crons             = ["0 0 12 * * *"]
    prohibit_overlap = true
    time_zone        = "Europe/Paris"
  }

  group "sdrot" {
    ephemeral_disk {
      size = 50
    }
    task "sdrot" {
      logs {
        max_files     = 5
        max_file_size = 5
      }
      driver = "raw_exec"
      config {
        command = "${NOMAD_TASK_DIR}/sentinel.sh"
      }
      template {
        destination = "local/sentinel.sh"
        perms       = "755"
        data        = <<-EOH
        #!/bin/sh
        # Storage read-stability sentinel. Runs as root via raw_exec
        # because /proc/sys/vm/drop_caches needs write privilege.
        # Non-zero exit => nomad marks the alloc failed.
        set -u
        TARGET=/usr/lib/os-release

        # 1. Read-stability of a static system file.
        a=$(sha256sum "$TARGET" | awk '{print $1}')
        echo 3 > /proc/sys/vm/drop_caches
        b=$(sha256sum "$TARGET" | awk '{print $1}')
        if [ "$a" != "$b" ]; then
          echo "FAIL read-stability $TARGET: $a vs $b" >&2
          exit 1
        fi

        # 2. Write + read-back of 1 MiB random data (catches write-side rot).
        T=$(mktemp /var/tmp/sdrot.XXXXXX)
        trap 'rm -f "$T"' EXIT
        head -c 1048576 /dev/urandom > "$T"
        sync
        echo 3 > /proc/sys/vm/drop_caches
        expected=$(sha256sum "$T" | awk '{print $1}')
        echo 3 > /proc/sys/vm/drop_caches
        got=$(sha256sum "$T" | awk '{print $1}')
        if [ "$expected" != "$got" ]; then
          echo "FAIL write-readback: $expected vs $got" >&2
          exit 1
        fi
        EOH
      }
      resources {
        cpu    = 50
        memory = 30
      }
    }
  }
}

Notes linking here

• ipfs get hangs on the empty-content leaf block

Permalink