Pre-calculate IPFS CID hashes for Pinata, Thirdweb, etc…

Pinning files on IPFS takes time. Wouldn’t it be great if we could calculate the CID hash ahead of time?

const myFile = fs.readFile(...)

const ipfsHash = calculateHash(myFile) // ← this is fast
const pinnedIpfsHash = await pinToIpfs(myFile) // ← this takes time

// ipfsHash == pinnedIpfsHash ← and they generate the same result

Turns out, you can do it simply with the modern ipfs library, Helia, with the unixfs module.

import { unixfs } from "@helia/unixfs"
import { BlackHoleBlockstore } from "blockstore-core/black-hole"

export const calculateCid = async (bytes: Uint8Array) => {
	const unixFs = unixfs({
	  blockstore: new BlackHoleBlockstore(),
	})
	
	const cid = await unixFs.addBytes(bytes)

	const cidString = cid.toString() // b45165...
}

This works, but it turns out that it doesn’t produce the same output as the cloud providers for files above 262kb. Probably something to do with chunking.

After lots of trial and error, I found the configuration that gives the same CIDs as Pinata and Thirdweb:

import { unixfs } from "@helia/unixfs"
import { BlackHoleBlockstore } from "blockstore-core/black-hole"
import { fixedSize } from "ipfs-unixfs-importer/chunker"
import { balanced } from "ipfs-unixfs-importer/layout"

export const calculateCid = async (bytes: Uint8Array) => {
	const unixFs = unixfs({
	  blockstore: new BlackHoleBlockstore(),
	})
	
	const cid = await unixFs.addBytes(bytes, {
	  cidVersion: 0,
	  rawLeaves: false,
	  leafType: "file",
	  layout: balanced({
	    maxChildrenPerNode: 174,
	  }),
	  chunker: fixedSize({
	    chunkSize: 262144,
	  }),
	})
	
	const cidv0 = cid.toV0().toString() // QmPK1s...
	const cidv1 = cid.toV1().toString() // b45165...
}

Now you can call function file throughout your app:

// an image
const image = fs.readFile(...)
const cid = await calculateCid(image)

// json
const jsonData = {
	test: 123
}
const textEncoder = new TextEncoder()
const jsonString = JSON.stringify(jsonData)
const uint8Array = textEncoder.encode(jsonString)
const cid = await calculateCid(uint8Array)

Hope this was helpful!

Pre-calculate IPFS CID hashes for Pinata, Thirdweb, etc…

Other posts

Tools