Debug Health of My Ipfs Swarm

what: the health of my ipfs swarm, debugged
tl;dr: on Oct 07 morning, gooseberry’s ipfs read a stale sda1 mount, its SMART-clean USB disk having disconnected at 05:56 and come back as sdb; 95 aborts on roddenberry’s USB disk hit a write, and it has 301 reallocated and 704 pending sectors.

Table of Contents

symptom
swarm peers
ipfs status
the failing block, allocation by allocation
reading a SMART report
gooseberry
roddenberry
Notes linking here
Permalink

symptom

for i in $(seq 10)
do
    curl -s http://home/stack-index/ | head -n 1
done

<!DOCTYPE html>
<!DOCTYPE html>
failed to resolve https://ipfs.konubinix.eu/p/bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4/: open /data/ipfs/blocks/SL/CIQOSJCTCCC7EPUHMBFK3LUJ2R7GL24TLTDR6BXLI3OXZ2GQY3UUSLY.data: input/output error: retrieval: no providers found for the CID (phase: path resolution)
<!DOCTYPE html>
failed to resolve https://ipfs.konubinix.eu/p/bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4/: open /data/ipfs/blocks/SL/CIQOSJCTCCC7EPUHMBFK3LUJ2R7GL24TLTDR6BXLI3OXZ2GQY3UUSLY.data: input/output error: retrieval: no providers found for the CID (phase: path resolution)
<!DOCTYPE html>
<!DOCTYPE html>
<!DOCTYPE html>
<!DOCTYPE html>
<!DOCTYPE html>

swarm peers

ipfs swarm peers

/ip4/192.168.2.14/tcp/4001/p2p/12D3KooWPus3L2qxYHBHnbjomrbU7VouFZiRCaLjwL86ExNz8yCJ
/ip4/192.168.2.20/tcp/4001/p2p/12D3KooWCRRAVNKmzLNpyXEHcasxT94jyaEjM3Pa4ZRphrbxtbxc
/ip4/192.168.2.21/tcp/4001/p2p/12D3KooWPdDokC7pjrKghj1Bsz3B9dfWbCgzjr4n61SHLW5FPKiP
/ip4/192.168.2.25/tcp/4001/p2p/12D3KooWEMZbfQu8BContbyftXAo5BCF5pJvfb1ASdF74Ns5rvAu

ipfs status

nd status ipfs | sed -n '/^Allocations/,$p'

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
402b25af  73403049  ipfs        251      run      running  1d19h ago  1d19h ago
8884e15b  601a455f  ipfs        251      run      running  3d18h ago  3d18h ago
a2c73623  b38a4e50  ipfs        251      run      running  3d18h ago  3d18h ago
bc1685a3  49e5a14a  ipfs        251      stop     lost     3d18h ago  1d19h ago
ffe12ed3  d12a8a26  ipfs        251      run      running  3d18h ago  3d18h ago

the failing block, allocation by allocation

for alloc in 402b25af 8884e15b a2c73623 ffe12ed3
do
    echo "${alloc}: $(timeout 30 nd alloc exec "${alloc}" \
        ipfs block stat --offline bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4 2>&1 \
        | head -n 1)"
done

402b25af: Key: bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4
8884e15b: Key: bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4
a2c73623: Key: bafybeihjerjrbbpsh2dwasvnv2e5i7tf5ojvzry7a3vunxl45dimn2kjf4
ffe12ed3: Error: error loading plugins: open /data/ipfs/config: input/output error

reading a SMART report

what

a USB disk, in this note, is a drive inside its USB enclosure, as its host sees it.

what

smartctl -H prints the drive’s own verdict, failing meaning failed or predicting its own failure within 24 hours¹.

what

an attribute fails once its normalized VALUE falls to its THRESH, and a failed Pre-fail attribute means failure is imminent².

what

attribute 5 counts the bad sectors found and reallocated³.

what

attribute 197 counts pending sectors, waiting to be reallocated after unrecoverable read errors⁴.

what

attribute 199 counts errors in data transfer over the interface cable⁵.

who

Backblaze recorded each day the SMART stats of the 67,814 drives in its data center, in 2016⁶.

what: Backblaze holds as a reason to investigate a raw value above zero in attribute 5, 187, 188, 197 or 198⁷.
what: in that fleet, one of those five was above zero on 76.7% of failed drives and 4.2% of working ones⁸.

gooseberry

the host of the failing allocation

nd alloc status ffe12ed3 | grep -E '^Node Name|^\*http'

Node Name           = gooseberry
*http  yes      192.168.2.21:27384

the journal

ssh gooseberry journalctl --dmesg --priority=err --no-pager \
    | grep -E 'Cannot enable|I/O error, dev sda|Remounting filesystem read-only' \
    | uniq

Oct 07 05:56:08 gooseberry kernel: usb 4-1-port4: Cannot enable. Maybe the USB cable is bad?
Oct 07 05:56:08 gooseberry kernel: I/O error, dev sda, sector 1182905456 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Oct 07 05:56:08 gooseberry kernel: EXT4-fs (sda1): Remounting filesystem read-only

the USB disk under the ipfs volume

volume_path="$(nd node status -verbose d12a8a26 | awk '$1 == "ipfs" {print $NF}')"
ssh gooseberry df --output=source,target "${volume_path}"

Filesystem     Mounted on
/dev/sdb1      /media/sam/bobux

ssh gooseberry journalctl --dmesg --no-pager | grep -E 'Attached SCSI disk'

Oct 03 09:24:20 gooseberry kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Oct 07 05:56:10 gooseberry kernel: sd 1:0:0:0: [sdb] Attached SCSI disk

nd alloc exec ffe12ed3 cat /proc/self/mountinfo < /dev/null \
    | awk '$5 == "/data/ipfs" {print $3, $5, $(NF-1)}'
ssh gooseberry lsblk --noheadings --output NAME,MAJ:MIN,SERIAL,LABEL < /dev/null

8:1 /data/ipfs /dev/sda1
sdb           8:16  X4KGT016T
└─sdb1        8:17             bobux
mmcblk0     179:0   0xab2757a1
├─mmcblk0p1 179:1              bootfs
└─mmcblk0p2 179:2              rootfs

what happened to its USB disk

ssh gooseberry "journalctl --dmesg --no-pager \
    --grep='hub_ext_port_status|USB disconnect|Direct-Access|EXT4-fs \(sd[ab]1\): (mounted|recovery)' \
    | uniq"

Oct 03 09:24:19 gooseberry kernel: scsi 0:0:0:0: Direct-Access     TOSHIBA  EXTERNAL_USB     0    PQ: 0 ANSI: 6
Oct 03 09:24:23 gooseberry kernel: EXT4-fs (sda1): mounted filesystem d137b942-35c5-491b-a0a5-f830185e937a r/w with ordered data mode. Quota mode: none.
Oct 03 09:24:58 gooseberry kernel: EXT4-fs (sda1): mounted filesystem d137b942-35c5-491b-a0a5-f830185e937a r/w with ordered data mode. Quota mode: none.
Oct 07 05:56:08 gooseberry kernel: hub 4-1:1.0: hub_ext_port_status failed (err = -71)
Oct 07 05:56:08 gooseberry kernel: usb 4-1.4: USB disconnect, device number 3
Oct 07 05:56:10 gooseberry kernel: scsi 1:0:0:0: Direct-Access     TOSHIBA  EXTERNAL_USB     0    PQ: 0 ANSI: 6
Oct 07 05:56:17 gooseberry kernel: EXT4-fs (sdb1): recovery complete
Oct 07 05:56:17 gooseberry kernel: EXT4-fs (sdb1): mounted filesystem d137b942-35c5-491b-a0a5-f830185e937a r/w with ordered data mode. Quota mode: none.

the history of USB disk errors

ssh gooseberry "journalctl --list-boots --no-pager \
    && journalctl --dmesg --boot=all --no-pager --grep='I/O error, dev sd|Cannot enable' \
    | uniq --skip-fields=5 --count"

IDX BOOT ID                          FIRST ENTRY                  LAST ENTRY
  0 05e894ad557a43e9b9d46d830fafc685 Sat 2026-10-03 09:24:19 CEST Wed 2026-10-07 07:31:29 CEST
      4 Oct 07 05:56:08 gooseberry kernel: usb 4-1-port4: Cannot enable. Maybe the USB cable is bad?
      1 Oct 07 05:56:08 gooseberry kernel: I/O error, dev sda, sector 1182905456 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2

the SMART of its USB disk

ssh gooseberry sudo -n smartctl -H -A -l error /dev/sdb \
    | grep -E 'overall-health|Reallocated|Pending|Uncorrectable|CRC|Errors Logged'

SMART overall-health self-assessment test result: PASSED
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
No Errors Logged

therefore: gooseberry’s drive reports no sector reallocated or pending, and no interface cable transfer error.

what the USB disconnect points at

ssh gooseberry "vcgencmd get_throttled \
    && lsusb -t \
    && journalctl --dmesg --no-pager \
       --grep='usb [34]-1(\.[0-9])?: (USB disconnect|reset|Product|SerialNumber)'"

throttled=0x0
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/2p, 480M
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 003.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 001: Dev 002, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 002: Dev 003, If 0, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 004.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 001: Dev 002, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 004: Dev 004, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
Oct 03 09:24:19 gooseberry kernel: usb 3-1: Product: USB2.1 Hub
Oct 03 09:24:19 gooseberry kernel: usb 4-1: Product: USB3.1 Hub
Oct 03 09:24:19 gooseberry kernel: usb 4-1.4: Product: EXTERNAL_USB
Oct 03 09:24:19 gooseberry kernel: usb 4-1.4: SerialNumber: 20241020009136F
Oct 03 09:24:19 gooseberry kernel: usb 3-1.2: Product: Eaton 3S
Oct 03 09:24:19 gooseberry kernel: usb 3-1.2: SerialNumber: Blank
Oct 07 05:56:08 gooseberry kernel: usb 4-1.4: USB disconnect, device number 3
Oct 07 05:56:09 gooseberry kernel: usb 4-1: reset SuperSpeed USB device number 2 using xhci-hcd
Oct 07 05:56:09 gooseberry kernel: usb 4-1.4: Product: EXTERNAL_USB
Oct 07 05:56:09 gooseberry kernel: usb 4-1.4: SerialNumber: 20241020009136F

ssh gooseberry "lsusb -v -s 4:2 2>/dev/null \
    | grep -E 'idVendor|idProduct|bmAttributes|Self Powered|Bus Powered|MaxPower|wHubCharacteristic|Power switching' ; \
    lsusb -v -s 3:2 2>/dev/null | grep -E 'idVendor|idProduct|Self Powered|Bus Powered|MaxPower'; \
    lsusb -s 3:3; lsusb -s 4:4"

  idVendor           0x05e3 Genesys Logic, Inc.
  idProduct          0x0626 Hub
    bmAttributes         0xe0
      Self Powered
    MaxPower                0mA
        bmAttributes           19
  idVendor           0x05e3 Genesys Logic, Inc.
  idProduct          0x0610 Hub
      Self Powered
    MaxPower              100mA
Bus 003 Device 003: ID 0463:ffff MGE UPS Systems UPS
Bus 004 Device 004: ID 0480:0906 Toshiba America Inc EXTERNAL_USB

what: both blocks above ran on Oct 07 before the hub swap that the block below shows, so they no longer run.

ssh gooseberry "uptime --since \
    && journalctl --dmesg --no-pager --grep='usb [34]-1: (Product|Manufacturer)'"

2026-10-07 08:37:40
Oct 07 08:37:44 gooseberry kernel: usb 3-1: Product: USB3 HUB
Oct 07 08:37:44 gooseberry kernel: usb 3-1: Manufacturer: Raspberry Pi
Oct 07 08:37:44 gooseberry kernel: usb 4-1: Product: USB3 HUB
Oct 07 08:37:44 gooseberry kernel: usb 4-1: Manufacturer: Raspberry Pi

what

in vcgencmd get_throttled, bit 16, 0x10000, set means undervoltage has occurred⁹.

therefore: gooseberry reads 0x0: no undervoltage is flagged as having occurred, over a window the documentation does not give.

what

the hub, one physical box as the user states, showed as two Genesys Logic hubs: its USB3.1 part on bus 4 and its USB2.1 part on bus 3, each on port 1.

what: the USB disk sat behind the hub’s USB3.1 part, the Eaton UPS behind the hub’s USB2.1 part.
therefore: at 05:56:09 the USB3.1 part was reset, while the UPS behind the USB2.1 part never disconnected.

therefore

nothing above rules out any of: gooseberry’s supply, its USB3 port, driver or controller firmware, the hub’s USB3.1 part, its cable and power supply, the USB disk cable, the enclosure’s USB bridge, the drive.

what

the hub ran on its own power supply, and its gooseberry-to-hub cable could not be taken off, as the user states.

nix-shell -p apcupsd --run "apcaccess -h 192.168.2.21:30745" \
    | grep -E '^(MODEL|STARTTIME|STATUS|LOTRANS|HITRANS|NUMXFERS|CUMONBATT)'

STARTTIME: 2026-10-03 07:28:48 +0000
MODEL    : Eaton 3S
STATUS   : ONLINE
LOTRANS  : 184.0 Volts
HITRANS  : 264.0 Volts
NUMXFERS : 0
CUMONBATT: 0 Seconds

what

NUMXFERS counts transfers to battery since apcupsd started, and the UPS switches to battery below LOTRANS or above HITRANS¹⁰.

therefore: the Eaton UPS saw no mains event that made it switch to battery since Oct 03, 05:56 included.

the swap experiments

how

change one part at a time: the hub with its cable first, for an official Raspberry Pi hub, then the USB disk cable.

why: the user bought the official Raspberry Pi hubs expecting them to be more reliable than this cheap one.
how: assuming a single fault, a reset after a change clears the part changed, and a quiet stretch well past the first 4 days points at it.
how: when the symptom shows, run the block below before any reboot, to tie it to a hub reset, and fill the row.
- what: gooseberry’s journal is set to Storage=volatile, as the block just below shows, and so is kept only in memory¹¹; a reboot wipes the reset that would explain the symptom.
- what: a hub reset always brings the symptom into the user’s day, as the user states.
  - therefore: with no symptom, a reboot loses no reset the table needs.

ssh gooseberry "grep -rH '^Storage' /etc/systemd/journald.conf /etc/systemd/journald.conf.d \
    /usr/lib/systemd/journald.conf.d 2>/dev/null; journalctl --header | grep -m1 'File path'"

/usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf:Storage=volatile
File path: /run/log/journal/1d3b95d611bf4cdda79c66034553e85a/system.journal

#	part changed	changed on	reset count	reset dates	read on
0	none, baseline	[2026-10-03 Sat]	1	[2026-10-07 Wed 05:56]	[2026-10-07 Wed]
1	official Raspberry Pi hub	[2026-10-07 Wed]
2	USB disk cable

ssh gooseberry "journalctl --dmesg --boot=all --no-pager --grep='usb 4-1: reset'"

Oct 07 05:56:09 gooseberry kernel: usb 4-1: reset SuperSpeed USB device number 2 using xhci-hcd

roddenberry

why: By mere chance, I took a look at roddenberry as well and found out it had troubles

nd node status | grep -E '^ID|roddenberry'

ID        Node Pool  DC       Name         Class   Drain  Eligibility  Status
601a455f  default    dc1      roddenberry  <none>  false  eligible     ready

what: roddenberry is nomad node 601a455f, which runs ipfs allocation 8884e15b in the ipfs status above.

the journal

ssh roddenberry "echo uas aborts: \$(journalctl --dmesg --no-pager | grep -c 'uas_eh_abort_handler') \
    && echo uas aborts by opcode of the aborted command: \
    && journalctl --dmesg --no-pager | grep -A1 'uas_eh_abort_handler' \
    | grep -o 'opcode=0x..' | sort | uniq --count \
    && echo usb disk resets per hour: \
    && journalctl --dmesg --no-pager --output=short-iso --grep='uas_eh_device_reset_handler start' \
    | cut -c1-13 | uniq --count"

uas aborts: 99
uas aborts by opcode of the aborted command:
      1 opcode=0x28
     95 opcode=0x2a
      1 opcode=0x35
      2 opcode=0xa1
usb disk resets per hour:
      4 2026-10-04T11
      2 2026-10-04T21
      7 2026-10-06T03
      1 2026-10-06T12

what: opcode 0x2a is WRITE(10)¹².
therefore: 95 of the 99 uas aborts on roddenberry’s USB disk hit a write, and the USB disk was reset 14 times since Oct 04.

the SMART of its USB disk

ssh roddenberry sudo -n smartctl -i -H -A -l error /dev/sda \
    | grep -E 'Device Model|overall-health|Reallocated|Pending|Uncorrectable|CRC|Errors Logged'

Device Model:     TOSHIBA MQ04ABD200
SMART overall-health self-assessment test result: PASSED
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       301
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       301
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       704
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
No Errors Logged

therefore: roddenberry’s drive has reallocated 301 bad sectors and holds 704 pending ones, while its interface cable carried no transfer error.
therefore: neither the drive’s verdict, PASSED, nor any attribute shown flags failure: attribute 5 still reads 100 against a threshold of 50, while its raw count is 301.
therefore: by Backblaze’s rule, roddenberry’s drive gives a reason to investigate, in the group that held 76.7% of Backblaze’s failed drives.

what to do about its drive

what

in Google’s fleet, drives were over 14 times likelier to fail within 60 days after their first reallocation, a count of the times a sector was reallocated¹³.

therefore: attribute 5 reads 301 on roddenberry, so at least one reallocation happened there.

what

there, drives were 16 times likelier to fail within 60 days after their first sector put on probation¹⁴.

therefore: probation resembles attribute 197’s pending sectors, an analogy the paper does not draw.

therefore

roddenberry’s drive sits in the reallocation group, and, if the analogy holds, in the probation group too.

how

copy what only roddenberry holds off the drive, then replace the drive, as the user settled.

why so young a drive gave way

the drive is shingled

what

a shingled, SMR, drive writes each track over part of the one before, like roof shingles, to fit more tracks on a platter¹⁵.

what: a conventional, CMR, drive writes tracks that do not overlap¹⁵.
- therefore: a CMR drive rewrites any block in place, touching nothing else.
what: on SMR, writing a track overwrites its neighbour, which must be rewritten too when it holds data¹⁵.
- therefore: rewriting one block may cost an SMR drive rewriting the tracks after it.
what: continuous writing of large amounts of data is noticeably slower than on CMR drives¹⁵.
what: a device-managed SMR drive hides this in its firmware and shows to the host as a plain drive¹⁵.

what

Toshiba lists drive-managed SMR among the MQ04 series’ features, for the MQ04ABD200 among others¹⁶.

what: smartctl reports roddenberry’s old drive, an MQ04ABD200, as zoned, device managed, as the block below shows.
therefore: bimbap’s drive is SMR, by its maker’s word and by what the drive itself reports.

ssh -n roddenberry sudo -n smartctl -i /dev/sda | grep -E 'Device Model|Rotation|Zoned'

Device Model:     TOSHIBA MQ04ABD200
Rotation Rate:    5400 rpm
Zoned Device:     Device managed zones

what

ipfs stores each block as a small file, so it writes many small files all over the disk.

therefore: on an SMR drive each of those writes may make the drive rewrite neighbouring tracks, which fits the aborted writes and resets the journal above shows.

what the drive and the logs record

ssh -n roddenberry sudo -n smartctl -A /dev/sda | grep -E 'Power_Cycle|Retract'

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       11

ssh -n roddenberry sudo -n smartctl -A /dev/sda | grep -E 'Power_On_Hours'
ssh -n roddenberry sudo -n tune2fs -l /dev/sda1 | grep -E 'Filesystem created'

  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       2556
Filesystem created:       Thu Jun 25 17:57:11 2026

what

the drive had run 2542 hours, about 106 days, when its bad sectors kept growing.

what: its filesystem was made on Jun 25, about as many hours ago as the drive has run, as the block above shows.
- therefore: the drive has run without a stop since about when its filesystem was made.
what: attribute 192 counts power-off or emergency head retracts¹⁷; it reads 11 on bimbap’s drive, against 14 power cycles in attribute 12, as the block above shows.
- therefore: about one retract per power cycle: nothing points at power cuts beyond the drive’s own power cycles, its hub having its own supply, as the user states.

what

smartd logs every attribute of the drive each 30 min since Oct 07, 07:43; the first block below prints the readings where attribute 5 or 197 changed.

what: loki keeps roddenberry’s kernel lines since Oct 07; the second block prints the disk’s resets and read errors, to set beside those readings.

ssh -n roddenberry cat /var/lib/smartmontools/attrlog.TOSHIBA_MQ04ABD200-54R8PHXFT.ata.csv \
    | awk -F'\t' '{ for (i = 2; i <= NF; i++) { split($i, a, ";"); if (a[1] == 5) r = a[3]; if (a[1] == 197) p = a[3] }
                    if (r p != last) print $1, "reallocated", r, "pending", p; last = r p }'

2026-10-07 07:43:49; reallocated 301 pending 704
2026-10-07 11:25:49; reallocated 309 pending 872
2026-10-07 11:55:51; reallocated 326 pending 1096
2026-10-07 12:25:49; reallocated 329 pending 1096
2026-10-07 13:55:49; reallocated 330 pending 1096
2026-10-07 14:25:50; reallocated 332 pending 1096
2026-10-07 14:55:50; reallocated 342 pending 1128
2026-10-07 15:56:04; reallocated 369 pending 1024
2026-10-07 16:25:49; reallocated 409 pending 832
2026-10-07 16:55:50; reallocated 438 pending 752
2026-10-07 17:55:50; reallocated 442 pending 752
2026-10-07 18:55:51; reallocated 529 pending 656
2026-10-07 20:25:50; reallocated 564 pending 680
2026-10-07 20:55:49; reallocated 565 pending 680
2026-10-07 22:25:49; reallocated 565 pending 688
2026-10-08 07:25:50; reallocated 581 pending 792
2026-10-08 07:55:50; reallocated 582 pending 736
2026-10-08 08:25:49; reallocated 594 pending 976
2026-10-08 20:25:50; reallocated 594 pending 984
2026-10-09 12:55:50; reallocated 594 pending 992

nix-shell -p grafana-loki --run "logcli --addr=http://192.168.2.14:9913 query --quiet --limit=20000 \
    --from=2026-10-06T00:00:00Z --to=2026-10-10T00:00:00Z --timezone=Local --forward \
    '{ns=\"kernel\", host=\"roddenberry\"}
     |~ \`uas_eh_device_reset_handler start|critical medium error|I/O error, dev sda\`'" \
    | sed -E 's/^([^ ]+) .*\} +/\1 /'

2026-10-07T11:25:43+02:00 scsi host0: uas_eh_device_reset_handler start
2026-10-07T11:32:14+02:00 scsi host0: uas_eh_device_reset_handler start
2026-10-07T11:32:15+02:00 I/O error, dev sda, sector 2378206336 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
2026-10-07T11:40:25+02:00 scsi host0: uas_eh_device_reset_handler start
2026-10-07T11:40:25+02:00 I/O error, dev sda, sector 2365614344 op 0x0:(READ) flags 0x80700 phys_seg 24 prio class 2
2026-10-07T11:40:25+02:00 I/O error, dev sda, sector 2365614544 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2
2026-10-07T11:43:00+02:00 scsi host0: uas_eh_device_reset_handler start
2026-10-07T11:45:48+02:00 scsi host0: uas_eh_device_reset_handler start
2026-10-08T13:11:16+02:00 critical medium error, dev sda, sector 58655808 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T13:13:51+02:00 critical medium error, dev sda, sector 58617632 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T13:19:04+02:00 critical medium error, dev sda, sector 58673312 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T13:19:05+02:00 critical medium error, dev sda, sector 58655808 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T13:19:06+02:00 critical medium error, dev sda, sector 58617632 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T19:39:36+02:00 critical medium error, dev sda, sector 36272608 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T19:52:38+02:00 critical medium error, dev sda, sector 36168448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T19:56:47+02:00 critical medium error, dev sda, sector 36272608 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-08T19:56:48+02:00 I/O error, dev sda, sector 36168544 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
2026-10-08T19:56:49+02:00 critical medium error, dev sda, sector 36168448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
2026-10-09T22:31:52+02:00 scsi host0: uas_eh_device_reset_handler start

what: a SMART extended self-test reads the whole surface and logs the first sector it could not read; the block below shows the one run on Oct 10.

ssh -n roddenberry sudo -n smartctl -l selftest /dev/sda | sed -n '/^Num/,$p'

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       00%      2563         35121192

what

the test failed first at LBA 35121192, below sectors 36168448 and 36272608, which the kernel could not read on Oct 08, as the errors block above shows.

what

the block below gives the drive’s lifetime temperatures and its shock count.

what: Toshiba gives the drive an operating range of 0 to 60 °C¹⁸.

ssh -n roddenberry sudo -n smartctl -l scttempsts /dev/sda | grep -E 'Lifetime|Over Temperature'
ssh -n roddenberry sudo -n smartctl -A /dev/sda | grep -E 'G-Sense'

Lifetime    Min/Max Temperature:     24/51 Celsius
Under/Over Temperature Limit Count:   0/0
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       1

what studies of drive fleets say

what

in Google’s fleet, drives failed more in their first three months than over their first year¹⁹.

what: there, high utilization raised failures only in the very young and very old age groups²⁰.
what: there, failures did not rise with average temperature, save at the very high end²¹.

what

Backblaze sees fewer early failures in its newer drives than in its 2013 ones²².

for the next disk

how

for the next ipfs disk, as the user settled:

how: pick a CMR drive, as its spec sheet states.
how: prefer a drive rated for 24x7 use and a yearly workload, as NAS drives are, to a laptop drive.
- what: Toshiba sells bimbap’s MQ04ABD200 as a client drive for laptops, rated for an MTTF of 600,000 h, with no workload nor hours of use a year stated¹⁸.
- what: Toshiba writes elsewhere that a client drive’s duty may be limited to 8 or 16 hours a day, and that its MTTF holds only within that duty²³.
- what: there, typical rated workloads reach 55 TB a year for client drives, 550 TB for enterprise ones²³.
- what: Western Digital sells its WD Red Plus, a CMR drive for NAS, for systems running 24x7, rated for 180 TB a year of workload and an MTBF of 1,000,000 h²⁴.

moving its cids to barbeuc

how

the move follows migrate an ipfs repo off a failing disk, its scripts tangled from there, with the old repo at media/sam/bimbap.ipfs, the new one at media/sam/barbeuc.ipfs, and the migration’s files in /media/sam/barbeuc/migration.

how: then the user switches roddenberry’s ipfs onto the barbeuc repo, repointing the ~/.ipfs link on roddenberry, as the user states.

the disk it moves to

what

barbeuc is to replace the drive under roddenberry’s ipfs node, holding only the cids the docs database allocates to roddenberry, the rest staying behind, as the user states.

what: barbeuc is a second USB disk on roddenberry, beside bimbap, the one holding the drive above, as the roddenberry-disks block below shows.
what: no pin may be added on the user’s nodes, and roddenberry’s node never runs garbage collection, as the user states.
- therefore: a cid is migrated once its blocks are in barbeuc’s repo; nothing has to hold them there.

nd alloc exec 8884e15b sh -c 'ipfs repo stat --human' < /dev/null

NumObjects: 2458743
RepoSize:   574 GB
StorageMax: 2.0 TB
RepoPath:   /data/ipfs
Version:    fs-repo@18

ssh roddenberry "lsblk --noheadings --output NAME,TRAN,MODEL,SIZE,LABEL,MOUNTPOINTS"

sda         usb    TOSHIBA MQ04ABD200    1.8T
└─sda1                                   1.8T bimbap  /media/sam/bimbap
sdb         usb    WDC WD20JDRW-11C7VS1  1.8T
└─sdb1                                   1.8T barbeuc /media/sam/barbeuc
mmcblk0     mmc                         29.8G
├─mmcblk0p1 mmc                          512M bootfs  /boot/firmware
└─mmcblk0p2 mmc                         29.3G rootfs  /

therefore: the whole repo weighs 574 GB, so roddenberry’s share of it fits barbeuc, a 2 TB disk lsblk lists as 1.8T.

the copy

how: the blocks below, in their order, make the new repo, check the sharding, cut the batches, measure the read order and the disks’ load, then start and follow the chain.

bash /var/run/user/1000/ipfshealth/init_repo.sh roddenberry /media/sam/barbeuc/.ipfs

bash /var/run/user/1000/ipfshealth/check_sharding.sh roddenberry /media/sam/bimbap/.ipfs \
    /media/sam/barbeuc/.ipfs

/repo/flatfs/shard/v1/next-to-last/2
/repo/flatfs/shard/v1/next-to-last/2
/repo/flatfs/shard/v1/next-to-last/2

what: the docs database allocates a cid to roddenberry in three places: a file’s own cid, a photo or video’s thumbnail, and its web version.

psql -h 192.168.2.5 -U postgres docs -At -c "
    select cid from file where allocations @> '[\"roddenberry\"]'
    union select thumbnail_cid from photovideo where thumbnail_allocations @> '[\"roddenberry\"]'
    union select web_cid from photovideo where web_allocations @> '[\"roddenberry\"]'" \
    | bash /var/run/user/1000/ipfshealth/cut_batches.sh roddenberry /media/sam/barbeuc/migration

0 new cids, from batch 111

bash /var/run/user/1000/ipfshealth/read_order.sh roddenberry /media/sam/bimbap/.ipfs/blocks \
    /media/sam/barbeuc/migration "${batch}"

hash_order: 800 files, 195 MB in 7460 ms, 26 MB/s
disk_order: 800 files, 201 MB in 4079 ms, 49 MB/s

therefore: read in disk order, the same kind of block files came off bimbap about twice as fast as in hash order.

bash /var/run/user/1000/ipfshealth/disk_load.sh roddenberry sda sdb

bash /var/run/user/1000/ipfshealth/start_chain.sh roddenberry /media/sam/barbeuc/migration \
    /media/sam/bimbap/.ipfs /media/sam/barbeuc/.ipfs

bash /var/run/user/1000/ipfshealth/chain_status.sh roddenberry /media/sam/barbeuc/migration

the rescue

how: the rescue tries the backup on bibou first, which must be mounted at /run/media/sam/bibou.

bash /var/run/user/1000/ipfshealth/rescue_broken.sh roddenberry /media/sam/barbeuc/migration \
    /media/sam/barbeuc/.ipfs clk backup docs retrieve

how fast it went

what

the chain’s log holds batches copied three ways, one after the other: by rsync, in name order; in disk order beside a dag stat of the batch before; in disk order with only the size check beside.

how: the block below times each batch from its listing to its copy’s end, sorting included, and sums bytes and time for each way.

ssh -n roddenberry cat /media/sam/barbeuc/migration/copy/all.log \
    | gawk 'function t(s) { gsub(/[-T:]/, " ", s); return mktime(substr(s, 1, 19)) }
            / [0-9]+ block files$/ { start = t($1); bytes = 0 }
            /^Total transferred file size/ { gsub(/,/, "", $5); bytes = $5; way = "1 rsync, name order" }
            /^copied [0-9]+ files, [0-9]+ bytes/ { bytes = $4; way = "2 disk order, dag stat beside" }
            / [0-9]+: copied/ && bytes > 0 {
                if ($0 ~ /every block file present|block files missing/) way = "3 disk order, size check beside"
                n[way]++; sum[way] += bytes; dur[way] += t($1) - start; bytes = 0 }
            END { for (w in n) printf "%s: %d batches, %.0f GB in %.1f h, %.0f MB/s\n",
                                     w, n[w], sum[w] / 1e9, dur[w] / 3600, sum[w] / 1e6 / dur[w] }' \
    | sort

1 rsync, name order: 20 batches, 184 GB in 5.1 h, 10 MB/s
2 disk order, dag stat beside: 11 batches, 100 GB in 2.2 h, 12 MB/s
3 disk order, size check beside: 22 batches, 212 GB in 2.3 h, 25 MB/s

therefore

from 10 MB/s in name order to 25 MB/s in disk order with only the size check beside, about 2.5 times as fast, sorting included.

therefore: sorting alone, with a dag stat beside, gained little, 12 MB/s: the dag stat starved the copy’s writes on barbeuc, as migrate an ipfs repo off a failing disk measures.

Notes linking here

migrate an ipfs repo off a failing disk

Permalink

« If the device reports failing health status, this means either that the device has already failed, or that it is predicting its own failure within the next 24 hours. » — primary, smartctl(8), smartmontools 7.4. ↩︎
« If the Normalized value is less than or equal to the Threshold value, then the Attribute is said to have failed. If the Attribute is a pre-failure Attribute, then disk failure is imminent. » — primary, smartctl(8), smartmontools 7.4. ↩︎
« Count of reallocated sectors. The raw value represents a count of the bad sectors that have been found and remapped. » — secondary, Wikipedia, S.M.A.R.T.; attributes are vendor specific, and the 13 pages of Toshiba’s MQ04AB manual, for roddenberry’s MQ04ABD200, read on 2026-10-07, define none; gooseberry’s drive model is not known. ↩︎
« Count of “unstable” sectors (waiting to be remapped, because of unrecoverable read errors). » — secondary, Wikipedia, S.M.A.R.T., same reason. ↩︎
« The count of errors in data transfer via the interface cable as determined by ICRC (Interface Cyclic Redundancy Check). » — secondary, Wikipedia, S.M.A.R.T., same reason. ↩︎
« Each day Backblaze records the SMART stats that are reported by the 67,814 hard drives we have spinning in our Sacramento data center. » — primary, Andy Klein, What SMART Stats Tell Us About Hard Drives, Backblaze, 2016-10-06. ↩︎
« For the last few years we’ve used the following five SMART stats as a means of helping determine if a drive is going to fail. » « SMART 5 Reallocated Sectors Count », « SMART 187 Reported Uncorrectable Errors », « SMART 188 Command Timeout », « SMART 197 Current Pending Sector Count », « SMART 198 Uncorrectable Sector Count » « When the RAW value for one of these five attributes is greater than zero, we have a reason to investigate. » — primary, Andy Klein, What SMART Stats Tell Us About Hard Drives, Backblaze, 2016-10-06. ↩︎
« Operational drives with one or more of our five SMART stats greater than zero: 4.2%. » « Failed drives with one or more of our five SMART stats greater than zero: 76.7%. » — primary, Andy Klein, What SMART Stats Tell Us About Hard Drives, Backblaze, 2016-10-06.
↩︎
« Returns the throttled state of the system. This is a bit pattern. A bit being set indicates the following meanings: » … « 16 | 0x10000 | Undervoltage has occurred » — primary, Raspberry Pi OS documentation, vcgencmd get_throttled. ↩︎
« NUMXFERS : Number of transfers to battery since apcupsd startup » « LOTRANS : Input line voltage below which UPS will switch to battery » « HITRANS : Input line voltage above which UPS will switch to battery » — primary, apcaccess(8), apcupsd 3.14.14. ↩︎
« journal log data will be stored only in memory » — primary, journald.conf(5), Storage=, the volatile value. ↩︎
« #define WRITE_10 0x2a » — primary, Linux include/scsi/scsi_proto.h. ↩︎
« When the drive’s logic believes that a sector is damaged (typically as a result of recurring soft errors or a hard error) it can remap the faulty sector number to a new physical sector drawn from a pool of spares. Reallocation counts reflect the number of times this has happened, and is seen as an indication of drive surface wear. » « After their first reallocation, drives are over 14 times more likely to fail within 60 days than drives without reallocation counts, making the critical threshold for this parameter also one. » — primary, Pinheiro, Weber and Barroso, Failure Trends in a Large Disk Drive Population, FAST'07, 2007, section 3.5.2. ↩︎
« Disk drives put suspect bad sectors “on probation” until they either fail permanently and are reallocated or continue to work without problems. » « The critical threshold for probational counts is also one: after the first event, drives are 16 times more likely to fail within 60 days than drives with zero probational counts. » — primary, same paper, section 3.5.4. ↩︎
« continuous writing of large amounts of data is noticeably slower than with CMR drives. » « shingled recording writes new tracks that overlap part of the previously written magnetic track, leaving the previous track narrower and allowing higher track density. » « writing to one track also overwrites an adjacent track. » « If adjacent tracks contain valid data, they must be rewritten as well. » « Conventional hard disk drives record data by writing non-overlapping concentric magnetic tracks » « A device-managed or drive-managed drive appears to the host identically to a non-shingled drive. » — secondary, Wikipedia, Shingled magnetic recording, read on 2026-10-09. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
« Drive-Managed SMR (Shingled magnetic recording) Technology », model numbers « MQ04ABD200 » and « MQ04ABF100 » — primary, MQ04 Series, Toshiba, read on 2026-10-09. ↩︎
« Number of power-off or emergency retract cycles. » — secondary, Wikipedia, S.M.A.R.T., attribute 192. ↩︎
« It is suitable for ultra-portable laptop and notebook PCs » « MTTF 600,000 h » « Temperature Operating 0 °C to 60 °C » « This drive uses 5V power only. 3.3V and 12V powe are not used. » — primary, MQ04AB manual, Toshiba, Rev. 01, Feb 2018, read on 2026-10-09. ↩︎ ↩︎
« our 3-month, 6-months and 1-year data points do seem to indicate a noticeable influence of infant mortality phenomena, with 1-year AFR dropping significantly from the AFR observed in the first three months. » — primary, Pinheiro, Weber and Barroso, same paper as above, section 3.1. ↩︎
« only very young and very old age groups appear to show the expected behavior. After the first year, the AFR of high utilization drives is at most moderately higher than that of low utilization drives. » — primary, same paper, section 3.3. ↩︎
« The figure shows that failures do not increase when the average temperature increases. » « Only at very high temperatures is there a slight reversal » — primary, same paper, section 3.4. ↩︎
« The 2013 data looks the closest to a true bathtub curve » « the 2021 data shows fewer early failures and a lower failure rate for more years » — primary, Backblaze, Are Hard Drives Getting Better?, read on 2026-10-09. ↩︎
« While business critical enterprise drives are designed for 24 hours per day, 7 days a week, 365 days a year (24/7/365 or 24/7) continuous operation, the operation duty of client drives may be limited, i.e. 8 or 16 hours per day for dedicated hard disk drives for desktop or laptop computers. » « Within the warranty period, a drive manufacturer estimates that a population of drives will not exceed a specified failure rate. Again, upon the condition that operating duty and other environmental constraints are adhered to. » « Typical rated workloads are up to 55 TB/year for client devices and up to 550 TB/year for enterprise/server drives. » — primary, Reliability of Enterprise Hard Disk Drives, Toshiba white paper, September 2015, sections 1.1, 1.3 and 2.2, read on 2026-10-10. ↩︎ ↩︎
« WD Red Plus hard drives are designed for systems that operate 24x7 » « Recording technology CMR » « Workload rate (TB/year) 180 » « MTBF (hours) 1,000,000 » « 12VDC ±5% (A, peak) » — primary, WD Red Plus brief, Western Digital, March 2025, read on 2026-10-09. ↩︎