Add docs on statediff and service dependencies and Ethereum upgrades
This commit is contained in:
commit
0288ee708c
130
plugeth-statediff-upgrade-guide.md
Normal file
130
plugeth-statediff-upgrade-guide.md
Normal file
@ -0,0 +1,130 @@
|
||||
# Upgrading plugeth-statediff for Ethereum fork
|
||||
|
||||
This guide covers the process of updating `plugeth-statediff` for a new Ethereum fork (e.g., Deneb/Cancun, Pectra, etc.).
|
||||
|
||||
See [project-dependencies.md](./project-dependencies.md) for the full dependency graph of all related projects.
|
||||
|
||||
## Overview
|
||||
|
||||
plugeth-statediff is the core state-diffing plugin that extracts Ethereum state changes. It depends on:
|
||||
- **plugeth**: The plugin framework (Geth fork)
|
||||
- **plugeth-utils**: Shared utilities for plugins
|
||||
|
||||
Integration tests additionally depend on:
|
||||
- **stack-orchestrator**: For deployment
|
||||
- **fixturenet-eth-stacks**: Stack definitions for local test networks
|
||||
- **system-tests**: For end-to-end testing
|
||||
- **eth-statediff-compliance**: For verification and benchmarking
|
||||
|
||||
## Update sequence
|
||||
|
||||
### Phase 1: Update utility tools to target new geth version
|
||||
|
||||
These tools depend on geth directly and should be updated first. They can be done in parallel:
|
||||
|
||||
1. **eth-testing** - Update to new geth version
|
||||
2. **eth-iterator-utils** - Update to new geth version
|
||||
3. **eth-dump-genblock** - Update to new geth version
|
||||
4. **eth-statediff-compliance** - Update to new geth version
|
||||
|
||||
These updates ensure testing and validation infrastructure is ready before updating the core plugin.
|
||||
|
||||
### Phase 2: Update core plugin stack
|
||||
|
||||
These should be done sequentially in this order:
|
||||
|
||||
#### 1. Update plugeth-utils
|
||||
|
||||
Start here as it's the base dependency for both plugeth and plugeth-statediff.
|
||||
|
||||
- Update to track new geth version
|
||||
- Test and release first
|
||||
|
||||
#### 2. Update plugeth
|
||||
|
||||
- Update plugeth-utils reference in go.mod
|
||||
- **Also update plugeth-utils in `wrapmain/go.mod`**
|
||||
- Rebase or update to target new geth version
|
||||
- Test and release
|
||||
|
||||
#### 3. Update plugeth-statediff
|
||||
|
||||
Main plugin update:
|
||||
|
||||
**Dependencies to update:**
|
||||
- Update plugeth reference
|
||||
- Update plugeth-utils reference
|
||||
|
||||
**Code changes:**
|
||||
- Adapt to any geth core/state API changes
|
||||
- Update state diffing logic if needed
|
||||
|
||||
**Testing requirements:**
|
||||
- Unit tests must pass
|
||||
- Integration tests (require stack-orchestrator and system-tests)
|
||||
- Compliance tests (using eth-statediff-compliance)
|
||||
|
||||
### Phase 3: Integration testing
|
||||
|
||||
Before releasing plugeth-statediff, ensure integration tests pass:
|
||||
|
||||
1. Update **stack-orchestrator** with any needed config changes
|
||||
2. Update **fixturenet-eth-stacks** if stack definitions need changes for the new fork
|
||||
3. Deploy local test network using fixturenet-eth-stacks
|
||||
4. Update **system-tests** if needed
|
||||
5. Run full integration test suite against the fixturenet
|
||||
6. Run **eth-statediff-compliance** benchmarks
|
||||
|
||||
## Known issues and workarounds
|
||||
|
||||
### Go plugin module name workaround
|
||||
|
||||
When updating plugeth, be aware of the Go plugin module naming issue:
|
||||
|
||||
- Issue: https://git.vdb.to/cerc-io/plugeth-statediff/issues/6
|
||||
- Fix: https://git.vdb.to/cerc-io/plugeth/commit/d730849b7e33a8c05d9f079ba2d5159cfb8cc66b
|
||||
|
||||
The module name must match exactly between plugin and host for Go's plugin system to work correctly.
|
||||
|
||||
## Release order
|
||||
|
||||
Recommended release order to minimize breaking changes:
|
||||
|
||||
1. **plugeth-utils** - Base dependency, release first
|
||||
2. **plugeth** - After plugeth-utils is released and tested
|
||||
3. **eth-testing** and other utility tools - Can be released in parallel
|
||||
4. **plugeth-statediff** - After plugeth and plugeth-utils are stable
|
||||
5. **Downstream services** - After plugeth-statediff is released (see project-dependencies.md for full list)
|
||||
|
||||
## Post-release updates
|
||||
|
||||
After releasing plugeth-statediff, the following projects will need updates:
|
||||
|
||||
**Immediate dependents:**
|
||||
- eth-statediff-service
|
||||
- ipld-eth-db-validator
|
||||
|
||||
**IPLD/StateDB layer:**
|
||||
- ipld-eth-statedb (can use `git patch/apply` from geth `core/state/*`)
|
||||
- ipfs-ethdb
|
||||
|
||||
See `project-dependencies.md` for the complete update sequence for all downstream services. For a
|
||||
concrete example, see the Notion milestone and tasks for the [Deneb/Cancun
|
||||
upgrade](https://www.notion.so/Plugeth-stack-Deneb-Cancun-updates-e90c2011baed48b093712e8ef52f9d6c).
|
||||
|
||||
## Checklist
|
||||
|
||||
Before considering the upgrade complete:
|
||||
|
||||
- [ ] plugeth-utils updated and released
|
||||
- [ ] plugeth updated (including wrapmain/go.mod) and released
|
||||
- [ ] plugeth-statediff updated with new dependency versions
|
||||
- [ ] Unit tests pass
|
||||
- [ ] fixturenet-plugeth stack successfully builds and deploys with updated versions
|
||||
- [ ] Integration tests pass (using fixturenet-plugeth stack via stack-orchestrator)
|
||||
- [ ] Compliance tests pass (eth-statediff-compliance)
|
||||
- [ ] System tests pass
|
||||
- [ ] Performance benchmarks reviewed
|
||||
- [ ] Release notes prepared
|
||||
- [ ] Downstream services identified for follow-up updates
|
||||
- [ ] Update these docs if needed
|
||||
272
project-dependencies.md
Normal file
272
project-dependencies.md
Normal file
@ -0,0 +1,272 @@
|
||||
# Ethereum statediff infrastructure projects
|
||||
|
||||
This document lists all projects involved in the Ethereum state-diffing infrastructure maintained by Vulcanize/Cerc. The infrastructure consists of 20 repositories spanning core plugins, utilities, IPLD/database layers, RPC servers, and supporting tools.
|
||||
|
||||
## Core plugin components
|
||||
|
||||
### [plugeth](https://git.vdb.to/cerc-io/plugeth)
|
||||
|
||||
Geth fork with plugin architecture for extending Ethereum client functionality without modifying core code.
|
||||
|
||||
**Dependencies:** none
|
||||
|
||||
**Used by:** plugeth-statediff
|
||||
|
||||
### [plugeth-utils](https://git.vdb.to/cerc-io/plugeth-utils)
|
||||
|
||||
Utility library providing shared functionality for plugeth plugins.
|
||||
|
||||
**Dependencies:** none
|
||||
|
||||
**Used by:** plugeth-statediff, plugeth
|
||||
|
||||
### [plugeth-statediff](https://git.vdb.to/cerc-io/plugeth-statediff)
|
||||
|
||||
Core state-diffing plugin that extracts Ethereum state changes between blocks as a plugeth plugin.
|
||||
|
||||
**Dependencies:**
|
||||
- plugeth
|
||||
- plugeth-utils
|
||||
|
||||
**Used by:**
|
||||
- eth-statediff-service
|
||||
- ipld-eth-db-validator
|
||||
- Integration tests (via stack-orchestrator)
|
||||
|
||||
## Utility tools
|
||||
|
||||
### [eth-testing](https://git.vdb.to/cerc-io/eth-testing)
|
||||
|
||||
Ethereum testing utilities and helpers.
|
||||
|
||||
**Dependencies:** geth directly
|
||||
|
||||
**Used by:**
|
||||
- eth-statediff-service
|
||||
- ipld-eth-state-snapshot
|
||||
|
||||
### [eth-iterator-utils](https://git.vdb.to/cerc-io/eth-iterator-utils)
|
||||
|
||||
Iterator utilities for traversing Ethereum state tries, including PrefixBoundIterator and SubtrieIterators.
|
||||
|
||||
**Dependencies:** geth directly
|
||||
|
||||
**Used by:** eth-ipfs-state-validator
|
||||
|
||||
### [eth-dump-genblock](https://git.vdb.to/cerc-io/eth-dump-genblock)
|
||||
|
||||
Tool to generate genesis blocks from Geth configuration files.
|
||||
|
||||
**Dependencies:** geth directly
|
||||
|
||||
### [eth-statediff-compliance](https://git.vdb.to/cerc-io/eth-statediff-compliance)
|
||||
|
||||
Tooling to verify correctness and benchmark performance of statediff results.
|
||||
|
||||
**Dependencies:** geth directly
|
||||
|
||||
**Used by:** plugeth-statediff integration tests
|
||||
|
||||
## IPLD and database abstraction layer
|
||||
|
||||
### [ipld-eth-db](https://git.vdb.to/cerc-io/ipld-eth-db)
|
||||
|
||||
Postgres database schemas and utilities for storing Ethereum IPLD objects.
|
||||
|
||||
**Dependencies:** none (defines schemas)
|
||||
|
||||
**Used by:**
|
||||
- ipld-eth-statedb
|
||||
- ipld-eth-server
|
||||
- All services that index or query Ethereum data
|
||||
|
||||
### [ipfs-ethdb](https://git.vdb.to/cerc-io/ipfs-ethdb)
|
||||
|
||||
Implementation of ethdb interfaces for Ethereum data stored in IPFS/Postgres.
|
||||
|
||||
**Dependencies:** none (interface definitions)
|
||||
|
||||
**Used by:**
|
||||
- ipld-eth-statedb
|
||||
- ipld-eth-db-validator
|
||||
- eth-ipfs-state-validator
|
||||
|
||||
### [ipld-eth-statedb](https://git.vdb.to/cerc-io/ipld-eth-statedb)
|
||||
|
||||
StateDB implementations for different access patterns (direct_by_leaf, trie_by_cid). Provides state access backed by IPFS/Postgres instead of LevelDB.
|
||||
|
||||
**Dependencies:**
|
||||
- plugeth-statediff (for state diffing)
|
||||
- ipfs-ethdb (for database access)
|
||||
|
||||
**Used by:**
|
||||
- eth-ipfs-state-validator
|
||||
- ipld-eth-server
|
||||
- ipld-eth-db-validator
|
||||
|
||||
## RPC server and validation
|
||||
|
||||
### [ipld-eth-server](https://git.vdb.to/cerc-io/ipld-eth-server)
|
||||
|
||||
JSON-RPC server providing Ethereum RPC endpoints backed by indexed IPLD objects in Postgres.
|
||||
|
||||
**Dependencies:**
|
||||
- eth-ipfs-state-validator
|
||||
- ipld-eth-statedb
|
||||
|
||||
**Used by:**
|
||||
- ipld-eth-db-validator (for verification)
|
||||
- chain-chunker (for verification)
|
||||
- External clients querying indexed data
|
||||
|
||||
### [eth-ipfs-state-validator](https://git.vdb.to/cerc-io/eth-ipfs-state-validator)
|
||||
|
||||
Validates completeness and correctness of IPFS-backed Ethereum state data.
|
||||
|
||||
**Dependencies:**
|
||||
- ipld-eth-statedb
|
||||
- eth-iterator-utils
|
||||
|
||||
**Used by:** ipld-eth-server
|
||||
|
||||
## Standalone services
|
||||
|
||||
### [ipld-eth-db-validator](https://git.vdb.to/cerc-io/ipld-eth-db-validator)
|
||||
|
||||
Validates indexed Ethereum IPLD objects stored in Postgres database.
|
||||
|
||||
**Dependencies:**
|
||||
- ipfs-ethdb
|
||||
- ipld-eth-server
|
||||
- ipld-eth-statedb
|
||||
- plugeth-statediff
|
||||
|
||||
### [eth-statediff-service](https://git.vdb.to/cerc-io/eth-statediff-service)
|
||||
|
||||
Standalone service for performing state diffing on a geth database.
|
||||
|
||||
**Dependencies:**
|
||||
- eth-testing
|
||||
- plugeth-statediff
|
||||
|
||||
**Used by:** chain-chunker
|
||||
|
||||
### [ipld-eth-state-snapshot](https://git.vdb.to/cerc-io/ipld-eth-state-snapshot)
|
||||
|
||||
Extracts Ethereum state at specific block heights for offline processing.
|
||||
|
||||
**Dependencies:** eth-testing
|
||||
|
||||
**Used by:** chain-chunker
|
||||
|
||||
### [chain-chunker](https://git.vdb.to/cerc-io/chain-chunker)
|
||||
|
||||
Orchestrates offline statediffing and data import pipeline.
|
||||
|
||||
**Dependencies:**
|
||||
- eth-statediff-service
|
||||
- ipld-eth-state-snapshot
|
||||
- ipld-eth-server (for verification)
|
||||
|
||||
## Supporting infrastructure
|
||||
|
||||
### [lighthouse](https://git.vdb.to/cerc-io/lighthouse)
|
||||
|
||||
Ethereum consensus client (beacon chain) used in test networks, forked to apply minor patches.
|
||||
|
||||
**Dependencies:** none (external dependency)
|
||||
|
||||
**Used by:** fixturenet-eth-stacks
|
||||
|
||||
### [stack-orchestrator](https://git.vdb.to/cerc-io/stack-orchestrator)
|
||||
|
||||
Deployment tool for setting up, building, and deploying Docker-based service stacks.
|
||||
|
||||
**Used by:** Integration tests for plugeth-statediff and other services
|
||||
|
||||
### [fixturenet-eth-stacks](https://git.vdb.to/cerc-io/fixturenet-eth-stacks)
|
||||
|
||||
Stack-orchestrator stack definitions for deploying local Ethereum testnets for development and integration testing. Contains two main stacks:
|
||||
- **fixturenet-eth**: Basic geth + lighthouse testnet
|
||||
- **fixturenet-plugeth**: Full state-diffing integration stack with plugeth, indexing, and RPC server
|
||||
|
||||
**Dependencies:**
|
||||
- stack-orchestrator (deployment tool)
|
||||
- lighthouse (consensus client)
|
||||
- plugeth (for fixturenet-plugeth stack)
|
||||
- plugeth-statediff (for fixturenet-plugeth stack)
|
||||
- ipld-eth-db (for fixturenet-plugeth stack)
|
||||
- ipld-eth-server (for fixturenet-plugeth stack)
|
||||
|
||||
**Used by:** Integration tests and development workflows for plugeth-statediff and related services
|
||||
|
||||
### [system-tests](https://git.vdb.to/cerc-io/system-tests)
|
||||
|
||||
End-to-end system integration tests.
|
||||
|
||||
**Used by:** Integration tests for plugeth-statediff
|
||||
|
||||
## Dependency graph overview
|
||||
|
||||
```
|
||||
Foundational layer:
|
||||
plugeth-utils → plugeth → plugeth-statediff
|
||||
|
||||
Database/IPLD layer:
|
||||
ipld-eth-db (schemas)
|
||||
ipfs-ethdb (interfaces) → ipld-eth-statedb
|
||||
|
||||
Server layer:
|
||||
eth-ipfs-state-validator + ipld-eth-statedb → ipld-eth-server
|
||||
|
||||
Services layer:
|
||||
plugeth-statediff + eth-testing → eth-statediff-service
|
||||
eth-testing → ipld-eth-state-snapshot
|
||||
eth-statediff-service + ipld-eth-state-snapshot + ipld-eth-server → chain-chunker
|
||||
|
||||
Validation:
|
||||
multiple deps → ipld-eth-db-validator
|
||||
|
||||
Utilities:
|
||||
geth → eth-testing, eth-iterator-utils, eth-dump-genblock, eth-statediff-compliance
|
||||
|
||||
Integration testing:
|
||||
lighthouse + plugeth + plugeth-statediff + ipld-eth-db + ipld-eth-server
|
||||
→ fixturenet-eth-stacks (deployed via stack-orchestrator)
|
||||
```
|
||||
|
||||
## Update order for Ethereum fork upgrades
|
||||
|
||||
When updating for a new Ethereum fork or other major change, follow this sequence:
|
||||
|
||||
1. **Utility tools** (can be done in parallel):
|
||||
- eth-testing
|
||||
- eth-iterator-utils
|
||||
- eth-dump-genblock
|
||||
- eth-statediff-compliance
|
||||
|
||||
2. **Core plugin stack** (sequential):
|
||||
- plugeth-utils
|
||||
- plugeth
|
||||
- plugeth-statediff
|
||||
|
||||
3. **IPLD/database layer**:
|
||||
- ipfs-ethdb
|
||||
- ipld-eth-statedb
|
||||
|
||||
4. **Validation and server**:
|
||||
- eth-ipfs-state-validator
|
||||
- ipld-eth-server
|
||||
|
||||
5. **Standalone services**:
|
||||
- ipld-eth-db-validator
|
||||
- eth-statediff-service
|
||||
- ipld-eth-state-snapshot
|
||||
- chain-chunker
|
||||
|
||||
6. **Supporting infrastructure**:
|
||||
- stack-orchestrator (config updates as needed)
|
||||
- fixturenet-eth-stacks (stack definition updates if needed)
|
||||
- system-tests
|
||||
|
||||
See [plugeth-statediff-upgrade-guide.md](./plugeth-statediff-upgrade-guide.md) for a more comprehensive guide.
|
||||
Loading…
Reference in New Issue
Block a user