From 0288ee708c341d765811a32637da538c921ef8c9 Mon Sep 17 00:00:00 2001
From: Roy Crihfield <roy@manteia.ltd>
Date: Mon, 10 Nov 2025 18:03:05 +0800
Subject: [PATCH] Add docs on statediff and service dependencies and Ethereum
 upgrades

---
 plugeth-statediff-upgrade-guide.md | 130 ++++++++++++++
 project-dependencies.md            | 272 +++++++++++++++++++++++++++++
 2 files changed, 402 insertions(+)
 create mode 100644 plugeth-statediff-upgrade-guide.md
 create mode 100644 project-dependencies.md

diff --git a/plugeth-statediff-upgrade-guide.md b/plugeth-statediff-upgrade-guide.md
new file mode 100644
index 0000000..848f9d6
--- /dev/null
+++ b/plugeth-statediff-upgrade-guide.md
@@ -0,0 +1,130 @@
+# Upgrading plugeth-statediff for Ethereum fork
+
+This guide covers the process of updating `plugeth-statediff` for a new Ethereum fork (e.g., Deneb/Cancun, Pectra, etc.).
+
+See [project-dependencies.md](./project-dependencies.md) for the full dependency graph of all related projects.
+
+## Overview
+
+plugeth-statediff is the core state-diffing plugin that extracts Ethereum state changes. It depends on:
+- **plugeth**: The plugin framework (Geth fork)
+- **plugeth-utils**: Shared utilities for plugins
+
+Integration tests additionally depend on:
+- **stack-orchestrator**: For deployment
+- **fixturenet-eth-stacks**: Stack definitions for local test networks
+- **system-tests**: For end-to-end testing
+- **eth-statediff-compliance**: For verification and benchmarking
+
+## Update sequence
+
+### Phase 1: Update utility tools to target new geth version
+
+These tools depend on geth directly and should be updated first. They can be done in parallel:
+
+1. **eth-testing** - Update to new geth version
+2. **eth-iterator-utils** - Update to new geth version
+3. **eth-dump-genblock** - Update to new geth version
+4. **eth-statediff-compliance** - Update to new geth version
+
+These updates ensure testing and validation infrastructure is ready before updating the core plugin.
+
+### Phase 2: Update core plugin stack
+
+These should be done sequentially in this order:
+
+#### 1. Update plugeth-utils
+
+Start here as it's the base dependency for both plugeth and plugeth-statediff.
+
+- Update to track new geth version
+- Test and release first
+
+#### 2. Update plugeth
+
+- Update plugeth-utils reference in go.mod
+- **Also update plugeth-utils in `wrapmain/go.mod`**
+- Rebase or update to target new geth version
+- Test and release
+
+#### 3. Update plugeth-statediff
+
+Main plugin update:
+
+**Dependencies to update:**
+- Update plugeth reference
+- Update plugeth-utils reference
+
+**Code changes:**
+- Adapt to any geth core/state API changes
+- Update state diffing logic if needed
+
+**Testing requirements:**
+- Unit tests must pass
+- Integration tests (require stack-orchestrator and system-tests)
+- Compliance tests (using eth-statediff-compliance)
+
+### Phase 3: Integration testing
+
+Before releasing plugeth-statediff, ensure integration tests pass:
+
+1. Update **stack-orchestrator** with any needed config changes
+2. Update **fixturenet-eth-stacks** if stack definitions need changes for the new fork
+3. Deploy local test network using fixturenet-eth-stacks
+4. Update **system-tests** if needed
+5. Run full integration test suite against the fixturenet
+6. Run **eth-statediff-compliance** benchmarks
+
+## Known issues and workarounds
+
+### Go plugin module name workaround
+
+When updating plugeth, be aware of the Go plugin module naming issue:
+
+- Issue: https://git.vdb.to/cerc-io/plugeth-statediff/issues/6
+- Fix: https://git.vdb.to/cerc-io/plugeth/commit/d730849b7e33a8c05d9f079ba2d5159cfb8cc66b
+
+The module name must match exactly between plugin and host for Go's plugin system to work correctly.
+
+## Release order
+
+Recommended release order to minimize breaking changes:
+
+1. **plugeth-utils** - Base dependency, release first
+2. **plugeth** - After plugeth-utils is released and tested
+3. **eth-testing** and other utility tools - Can be released in parallel
+4. **plugeth-statediff** - After plugeth and plugeth-utils are stable
+5. **Downstream services** - After plugeth-statediff is released (see project-dependencies.md for full list)
+
+## Post-release updates
+
+After releasing plugeth-statediff, the following projects will need updates:
+
+**Immediate dependents:**
+- eth-statediff-service
+- ipld-eth-db-validator
+
+**IPLD/StateDB layer:**
+- ipld-eth-statedb (can use `git patch/apply` from geth `core/state/*`)
+- ipfs-ethdb
+
+See `project-dependencies.md` for the complete update sequence for all downstream services.  For a
+concrete example, see the Notion milestone and tasks for the [Deneb/Cancun
+upgrade](https://www.notion.so/Plugeth-stack-Deneb-Cancun-updates-e90c2011baed48b093712e8ef52f9d6c).
+
+## Checklist
+
+Before considering the upgrade complete:
+
+- [ ] plugeth-utils updated and released
+- [ ] plugeth updated (including wrapmain/go.mod) and released
+- [ ] plugeth-statediff updated with new dependency versions
+- [ ] Unit tests pass
+- [ ] fixturenet-plugeth stack successfully builds and deploys with updated versions
+- [ ] Integration tests pass (using fixturenet-plugeth stack via stack-orchestrator)
+- [ ] Compliance tests pass (eth-statediff-compliance)
+- [ ] System tests pass
+- [ ] Performance benchmarks reviewed
+- [ ] Release notes prepared
+- [ ] Downstream services identified for follow-up updates
+- [ ] Update these docs if needed
diff --git a/project-dependencies.md b/project-dependencies.md
new file mode 100644
index 0000000..1d1b99e
--- /dev/null
+++ b/project-dependencies.md
@@ -0,0 +1,272 @@
+# Ethereum statediff infrastructure projects
+
+This document lists all projects involved in the Ethereum state-diffing infrastructure maintained by Vulcanize/Cerc. The infrastructure consists of 20 repositories spanning core plugins, utilities, IPLD/database layers, RPC servers, and supporting tools.
+
+## Core plugin components
+
+### [plugeth](https://git.vdb.to/cerc-io/plugeth)
+
+Geth fork with plugin architecture for extending Ethereum client functionality without modifying core code.
+
+**Dependencies:** none
+
+**Used by:** plugeth-statediff
+
+### [plugeth-utils](https://git.vdb.to/cerc-io/plugeth-utils)
+
+Utility library providing shared functionality for plugeth plugins.
+
+**Dependencies:** none
+
+**Used by:** plugeth-statediff, plugeth
+
+### [plugeth-statediff](https://git.vdb.to/cerc-io/plugeth-statediff)
+
+Core state-diffing plugin that extracts Ethereum state changes between blocks as a plugeth plugin.
+
+**Dependencies:**
+- plugeth
+- plugeth-utils
+
+**Used by:**
+- eth-statediff-service
+- ipld-eth-db-validator
+- Integration tests (via stack-orchestrator)
+
+## Utility tools
+
+### [eth-testing](https://git.vdb.to/cerc-io/eth-testing)
+
+Ethereum testing utilities and helpers.
+
+**Dependencies:** geth directly
+
+**Used by:**
+- eth-statediff-service
+- ipld-eth-state-snapshot
+
+### [eth-iterator-utils](https://git.vdb.to/cerc-io/eth-iterator-utils)
+
+Iterator utilities for traversing Ethereum state tries, including PrefixBoundIterator and SubtrieIterators.
+
+**Dependencies:** geth directly
+
+**Used by:** eth-ipfs-state-validator
+
+### [eth-dump-genblock](https://git.vdb.to/cerc-io/eth-dump-genblock)
+
+Tool to generate genesis blocks from Geth configuration files.
+
+**Dependencies:** geth directly
+
+### [eth-statediff-compliance](https://git.vdb.to/cerc-io/eth-statediff-compliance)
+
+Tooling to verify correctness and benchmark performance of statediff results.
+
+**Dependencies:** geth directly
+
+**Used by:** plugeth-statediff integration tests
+
+## IPLD and database abstraction layer
+
+### [ipld-eth-db](https://git.vdb.to/cerc-io/ipld-eth-db)
+
+Postgres database schemas and utilities for storing Ethereum IPLD objects.
+
+**Dependencies:** none (defines schemas)
+
+**Used by:**
+- ipld-eth-statedb
+- ipld-eth-server
+- All services that index or query Ethereum data
+
+### [ipfs-ethdb](https://git.vdb.to/cerc-io/ipfs-ethdb)
+
+Implementation of ethdb interfaces for Ethereum data stored in IPFS/Postgres.
+
+**Dependencies:** none (interface definitions)
+
+**Used by:**
+- ipld-eth-statedb
+- ipld-eth-db-validator
+- eth-ipfs-state-validator
+
+### [ipld-eth-statedb](https://git.vdb.to/cerc-io/ipld-eth-statedb)
+
+StateDB implementations for different access patterns (direct_by_leaf, trie_by_cid). Provides state access backed by IPFS/Postgres instead of LevelDB.
+
+**Dependencies:**
+- plugeth-statediff (for state diffing)
+- ipfs-ethdb (for database access)
+
+**Used by:**
+- eth-ipfs-state-validator
+- ipld-eth-server
+- ipld-eth-db-validator
+
+## RPC server and validation
+
+### [ipld-eth-server](https://git.vdb.to/cerc-io/ipld-eth-server)
+
+JSON-RPC server providing Ethereum RPC endpoints backed by indexed IPLD objects in Postgres.
+
+**Dependencies:**
+- eth-ipfs-state-validator
+- ipld-eth-statedb
+
+**Used by:**
+- ipld-eth-db-validator (for verification)
+- chain-chunker (for verification)
+- External clients querying indexed data
+
+### [eth-ipfs-state-validator](https://git.vdb.to/cerc-io/eth-ipfs-state-validator)
+
+Validates completeness and correctness of IPFS-backed Ethereum state data.
+
+**Dependencies:**
+- ipld-eth-statedb
+- eth-iterator-utils
+
+**Used by:** ipld-eth-server
+
+## Standalone services
+
+### [ipld-eth-db-validator](https://git.vdb.to/cerc-io/ipld-eth-db-validator)
+
+Validates indexed Ethereum IPLD objects stored in Postgres database.
+
+**Dependencies:**
+- ipfs-ethdb
+- ipld-eth-server
+- ipld-eth-statedb
+- plugeth-statediff
+
+### [eth-statediff-service](https://git.vdb.to/cerc-io/eth-statediff-service)
+
+Standalone service for performing state diffing on a geth database.
+
+**Dependencies:**
+- eth-testing
+- plugeth-statediff
+
+**Used by:** chain-chunker
+
+### [ipld-eth-state-snapshot](https://git.vdb.to/cerc-io/ipld-eth-state-snapshot)
+
+Extracts Ethereum state at specific block heights for offline processing.
+
+**Dependencies:** eth-testing
+
+**Used by:** chain-chunker
+
+### [chain-chunker](https://git.vdb.to/cerc-io/chain-chunker)
+
+Orchestrates offline statediffing and data import pipeline.
+
+**Dependencies:**
+- eth-statediff-service
+- ipld-eth-state-snapshot
+- ipld-eth-server (for verification)
+
+## Supporting infrastructure
+
+### [lighthouse](https://git.vdb.to/cerc-io/lighthouse)
+
+Ethereum consensus client (beacon chain) used in test networks, forked to apply minor patches.
+
+**Dependencies:** none (external dependency)
+
+**Used by:** fixturenet-eth-stacks
+
+### [stack-orchestrator](https://git.vdb.to/cerc-io/stack-orchestrator)
+
+Deployment tool for setting up, building, and deploying Docker-based service stacks.
+
+**Used by:** Integration tests for plugeth-statediff and other services
+
+### [fixturenet-eth-stacks](https://git.vdb.to/cerc-io/fixturenet-eth-stacks)
+
+Stack-orchestrator stack definitions for deploying local Ethereum testnets for development and integration testing. Contains two main stacks:
+- **fixturenet-eth**: Basic geth + lighthouse testnet
+- **fixturenet-plugeth**: Full state-diffing integration stack with plugeth, indexing, and RPC server
+
+**Dependencies:**
+- stack-orchestrator (deployment tool)
+- lighthouse (consensus client)
+- plugeth (for fixturenet-plugeth stack)
+- plugeth-statediff (for fixturenet-plugeth stack)
+- ipld-eth-db (for fixturenet-plugeth stack)
+- ipld-eth-server (for fixturenet-plugeth stack)
+
+**Used by:** Integration tests and development workflows for plugeth-statediff and related services
+
+### [system-tests](https://git.vdb.to/cerc-io/system-tests)
+
+End-to-end system integration tests.
+
+**Used by:** Integration tests for plugeth-statediff
+
+## Dependency graph overview
+
+```
+Foundational layer:
+  plugeth-utils → plugeth → plugeth-statediff
+
+Database/IPLD layer:
+  ipld-eth-db (schemas)
+  ipfs-ethdb (interfaces) → ipld-eth-statedb
+
+Server layer:
+  eth-ipfs-state-validator + ipld-eth-statedb → ipld-eth-server
+
+Services layer:
+  plugeth-statediff + eth-testing → eth-statediff-service
+  eth-testing → ipld-eth-state-snapshot
+  eth-statediff-service + ipld-eth-state-snapshot + ipld-eth-server → chain-chunker
+
+Validation:
+  multiple deps → ipld-eth-db-validator
+
+Utilities:
+  geth → eth-testing, eth-iterator-utils, eth-dump-genblock, eth-statediff-compliance
+
+Integration testing:
+  lighthouse + plugeth + plugeth-statediff + ipld-eth-db + ipld-eth-server
+    → fixturenet-eth-stacks (deployed via stack-orchestrator)
+```
+
+## Update order for Ethereum fork upgrades
+
+When updating for a new Ethereum fork or other major change, follow this sequence:
+
+1. **Utility tools** (can be done in parallel):
+   - eth-testing
+   - eth-iterator-utils
+   - eth-dump-genblock
+   - eth-statediff-compliance
+
+2. **Core plugin stack** (sequential):
+   - plugeth-utils
+   - plugeth
+   - plugeth-statediff
+
+3. **IPLD/database layer**:
+   - ipfs-ethdb
+   - ipld-eth-statedb
+
+4. **Validation and server**:
+   - eth-ipfs-state-validator
+   - ipld-eth-server
+
+5. **Standalone services**:
+   - ipld-eth-db-validator
+   - eth-statediff-service
+   - ipld-eth-state-snapshot
+   - chain-chunker
+
+6. **Supporting infrastructure**:
+   - stack-orchestrator (config updates as needed)
+   - fixturenet-eth-stacks (stack definition updates if needed)
+   - system-tests
+
+See [plugeth-statediff-upgrade-guide.md](./plugeth-statediff-upgrade-guide.md) for a more comprehensive guide.