A service that indexes blobs on the Beacon chain

Go to file

Roy Crihfield b40ddade28 Implement Blobscan scraper		2024-06-07 19:31:03 +08:00
.gitea/workflows	Fix CI tests (#1 )	2024-05-30 11:12:24 +00:00
cmd	Implement Blobscan scraper	2024-06-07 19:31:03 +08:00
pkg	Implement Blobscan scraper	2024-06-07 19:31:03 +08:00
scripts	Fix CI tests (#1 )	2024-05-30 11:12:24 +00:00
test	Implement Blobscan scraper	2024-06-07 19:31:03 +08:00
.dockerignore	Initial service	2024-05-30 13:24:45 +08:00
doc.go	Initial service	2024-05-30 13:24:45 +08:00
Dockerfile	Implement Blobscan scraper	2024-06-07 19:31:03 +08:00
go.mod	Initial service	2024-05-30 13:24:45 +08:00
go.sum	Initial service	2024-05-30 13:24:45 +08:00
LICENSE	Add README, LICENSE	2024-05-30 13:24:45 +08:00
README.md	Implement Blobscan scraper	2024-06-07 19:31:03 +08:00

README.md

eth-blob-indexer

Tools to index Ethereum blob data.

beacon-blob-collector

Service that indexes all newly committed blobs on the Ethereum Beacon chain.

Build:

$ go build ./cmd/beacon-blob-collector

Run:

$ beacon-blob-collector \
    --beacon-addr=http://fixturenet-eth-lighthouse-1:8001 \
    --redis-addr=eth-blob-db:6379 \
    --log-level=debug
May 29 17:11:54.397 INF Starting indexer beacon=http://fixturenet-eth-lighthouse-1:8001 redis=eth-blob-db:6379
May 29 17:11:54.398 DBG Subscribing to event stream endpoint="http://fixturenet-eth-lighthouse-1:8001/eth/v1/events?topics=blob_sidecar"
May 29 17:11:54.398 DBG Subscribing to event stream endpoint="http://fixturenet-eth-lighthouse-1:8001/eth/v1/events?topics=head"
May 29 17:11:55.058 DBG Received head event slot=153 block=0x838ca2e262da142b7f0b3aa5d0f3472a7e33d52e597aa0ab9cad609f2bb0fb87
...
May 29 17:16:13.062 DBG Received head event slot=239 block=0x0015e9ce0b104935c05abe49dac1e83721b339aa61954b1128dd178800e3bad7
May 29 17:16:16.070 DBG Received blob_sidecar event slot=240 block=0x304d37b405fb99913a9e2cb543080d80fb08871ca3c7d581e88b460c3678fa58 vhash=0x01265315cda2c869191467c60f7374dc80be8be32b8117cd780655014db819f6
May 29 17:16:16.098 DBG Received head event slot=240 block=0x304d37b405fb99913a9e2cb543080d80fb08871ca3c7d581e88b460c3678fa58
May 29 17:16:16.098 DBG Fetching blobs endpoint=http://fixturenet-eth-lighthouse-1:8001/eth/v1/beacon/blob_sidecars/240
May 29 17:16:16.107 INF Storing blob vhash=0x01265315cda2c869191467c60f7374dc80be8be32b8117cd780655014db819f6

blobscan-scraper

Service that scrapes a Blobscan instance for ranges of past blobs. Useful to backfill data no longer available from a Beacon node (assuming Blobscan has retained the data).

Build:

$ go build ./cmd/blobscan-scraper

Run:

$ blobscan-scraper --redis-addr localhost:57889 --log-level debug --block-start 20032451
Jun  7 18:11:01.619 INF Starting indexer blobscan=https://api.blobscan.com redis=localhost:57889
Jun  7 18:11:01.619 DBG Fetching block endpoint="https://api.blobscan.com/blocks?sort=asc&startBlock=20032451&endBlock=20032476"
Jun  7 18:11:03.115 DBG Processing block block=20032451
Jun  7 18:11:03.115 DBG Fetching blob endpoint=https://api.blobscan.com/blobs/0x0126e8b373caaba7556c5f410efc445c64235ef048354467a3aaf97588ca8490
Jun  7 18:11:04.665 INF Storing blob vhash=0x0126e8b373caaba7556c5f410efc445c64235ef048354467a3aaf97588ca8490
Jun  7 18:11:04.721 DBG Fetching blob endpoint=https://api.blobscan.com/blobs/0x01b57cd55c617e40af61a916cbafbd596646c70f69d4685383dfd67966aeadb3
Jun  7 18:11:05.726 INF Storing blob vhash=0x01b57cd55c617e40af61a916cbafbd596646c70f69d4685383dfd67966aeadb3
Jun  7 18:11:05.750 DBG Fetching blob endpoint=https://api.blobscan.com/blobs/0x01c7ffd0638a0d4b1774668813b7d8740de5ccfaaaac83c333f63b8a7ffa695e
Jun  7 18:11:06.663 INF Storing blob vhash=0x01c7ffd0638a0d4b1774668813b7d8740de5ccfaaaac83c333f63b8a7ffa695e
Jun  7 18:11:06.671 DBG Processing block block=20032453
Jun  7 18:11:06.671 DBG Fetching blob endpoint=https://api.blobscan.com/blobs/0x01badc50474c8e8353e4becf87147f14288198354948ef13a4b461f28624e87a