Ipfs Get Hangs on the Empty-Content Leaf Block

Sibling note to investigate an ipfs issue leads to finding out corrupted sd card — once the SD-card root cause was fixed and a fresh ipfs get retried, the download still hung. This is what was actually blocking it.

Symptom

After re-flashing the failed node and restarting all ipfs jobs, a fresh ipfs get bafybei… (the same ~333 MiB DAG as before) still hangs partway through. ipfs bitswap stat on the requesting node:

bitswap status
    blocks received: 4
    blocks sent:     0
    wantlist [1 keys]
        bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku
    partners [0]

On all four peer nodes, ipfs block stat --offline for that CID returns Size: 0 and ipfs block get produces 0 bytes with exit 0. First instinct is “block is corrupted / missing on every node” — that turns out to be wrong.

That CID is the canonical empty-content block

bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku decodes to CIDv1 + raw codec (0x55) wrapping sha256 of zero bytes (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). It is the well-known empty-content CID and IPFS docs use it as the canonical example. So the four peers each returning a 0-byte block under this CID were correct, not corrupted.

Why the get still hangs

Even though every peer has the block and is willing to serve it, bitswap on the requesting node did not retrieve it: partners [0] stayed at 0, and the want stayed in the list indefinitely. The DAG walker is stuck on this one leaf and never advances. Possible mechanisms (couldn’t pin down a single upstream issue):

• the bitswap server side treats a 0-byte payload as nothing-to-send and skips the HAVE/BLOCK exchange, or
• the receiving side discards the empty payload before counting it as a delivery, or
• the local blockservice short-circuits this constant CID in a way that doesn’t satisfy the bitswap want.

Closest references found:

• IPFS troubleshooting page uses this exact CID as the example for “ipfs get hangs with no output”:

https://docs.ipfs.tech/how-to/troubleshooting-kubo/

• File-transfer troubleshooting page, same CID:

https://docs.ipfs.tech/how-to/troubleshoot-file-transfers/

• Empty-file CID handling differences across clients:

https://github.com/ipfs-shipyard/py-ipfs-http-client/issues/293

• js-ipfs behaviour returning the empty-file CID even for non-empty inputs in certain configurations:

https://github.com/ipfs/js-ipfs/issues/1518

• Bitswap protocol spec (HAVE / WANT-BLOCK / payload framing):

https://specs.ipfs.tech/bitswap-protocol/

None of these is the precise “bitswap can’t retrieve a 0-byte block” bug, but together they document that the empty CID is a known quirky constant and that ipfs get can sit forever on it.

Workaround (one line)

The empty-content block is deterministic — every IPFS node can produce it locally with no network involved — so just put it directly into the requesting node’s own datastore:

printf '' | ipfs block put --cid-codec=raw --mhtype=sha2-256

Expected output:

bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku

After this, the blockservice satisfies the want from the local store and bitswap moves on. In our case the stuck get went from blocks received: 4 / data received: 14481 to blocks received: 413 / data received: 11191976 within seconds.

Older kubo without --cid-codec:

echo -n | ipfs block put --format=raw          # legacy flag name
ipfs add --raw-leaves --pin=false /dev/null    # also produces the same CID

When to suspect this rather than “data is missing”

• ipfs bitswap stat wantlist contains exactly that one CID (bafkreihdwd…hqu)
• partners [0] persists despite peers in ipfs swarm peers actually having the block (verify with ipfs block get <cid> | wc -c on each peer → expect 0 with exit code 0)
• The get had been progressing before, and most of the DAG is already in the requesting node’s local store from a prior partial fetch (so this is the only outstanding leaf)

If the stuck CID is anything other than bafkreihdwd…hqu, this note does not apply; check the SD-card investigation for the general bitswap-stall diagnosis flow.

Prevention

Cheapest mitigation: pre-seed every node with the empty-content block at provisioning time. It’s a constant, costs one datastore entry, and forever-immunises that node from this particular hang.

Add to the kubo start script (e.g. nomad/files/kubo/start.sh, after ipfs daemon is up and reachable — easiest done as a one-shot ipfs block put via the local API after a short sleep, or via a sidecar that polls the API and seeds the block once):

printf '' | ipfs block put --cid-codec=raw --mhtype=sha2-256

Doing this on every node in the private swarm also means each peer can locally satisfy its own DAG walker if it ever wants the empty leaf — no bitswap round-trip needed at all.

Permalink