2019-11-26 23:54:46 +00:00
|
|
|
use crate::chunked_vector::{
|
|
|
|
store_updated_vector, BlockRoots, HistoricalRoots, RandaoMixes, StateRoots,
|
|
|
|
};
|
2020-11-17 09:10:53 +00:00
|
|
|
use crate::config::{OnDiskStoreConfig, StoreConfig};
|
2021-07-06 02:38:53 +00:00
|
|
|
use crate::forwards_iter::{HybridForwardsBlockRootsIterator, HybridForwardsStateRootsIterator};
|
2021-07-09 06:15:32 +00:00
|
|
|
use crate::impls::{
|
|
|
|
beacon_block_as_kv_store_op,
|
|
|
|
beacon_state::{get_full_state, store_full_state},
|
|
|
|
};
|
2019-12-06 03:29:06 +00:00
|
|
|
use crate::iter::{ParentRootBlockIterator, StateRootsIterator};
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
use crate::leveldb_store::BytesKey;
|
2020-06-16 01:34:04 +00:00
|
|
|
use crate::leveldb_store::LevelDB;
|
|
|
|
use crate::memory_store::MemoryStore;
|
2020-09-30 02:36:07 +00:00
|
|
|
use crate::metadata::{
|
2020-11-17 09:10:53 +00:00
|
|
|
CompactionTimestamp, PruningCheckpoint, SchemaVersion, COMPACTION_TIMESTAMP_KEY, CONFIG_KEY,
|
|
|
|
CURRENT_SCHEMA_VERSION, PRUNING_CHECKPOINT_KEY, SCHEMA_VERSION_KEY, SPLIT_KEY,
|
2020-09-30 02:36:07 +00:00
|
|
|
};
|
2020-02-10 00:30:21 +00:00
|
|
|
use crate::metrics;
|
2019-11-26 23:54:46 +00:00
|
|
|
use crate::{
|
2020-06-16 01:34:04 +00:00
|
|
|
get_key_for_col, DBColumn, Error, ItemStore, KeyValueStoreOp, PartialBeaconState, StoreItem,
|
|
|
|
StoreOp,
|
2019-11-26 23:54:46 +00:00
|
|
|
};
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
use leveldb::iterator::LevelDBIterator;
|
2020-02-10 00:30:21 +00:00
|
|
|
use lru::LruCache;
|
|
|
|
use parking_lot::{Mutex, RwLock};
|
2020-08-26 00:01:06 +00:00
|
|
|
use slog::{debug, error, info, trace, warn, Logger};
|
2019-11-26 23:54:46 +00:00
|
|
|
use ssz::{Decode, Encode};
|
2019-12-06 03:29:06 +00:00
|
|
|
use ssz_derive::{Decode, Encode};
|
|
|
|
use state_processing::{
|
|
|
|
per_block_processing, per_slot_processing, BlockProcessingError, BlockSignatureStrategy,
|
|
|
|
SlotProcessingError,
|
|
|
|
};
|
2019-11-26 23:54:46 +00:00
|
|
|
use std::convert::TryInto;
|
2019-12-06 07:52:11 +00:00
|
|
|
use std::marker::PhantomData;
|
2019-11-26 23:54:46 +00:00
|
|
|
use std::path::Path;
|
|
|
|
use std::sync::Arc;
|
2020-11-17 09:10:53 +00:00
|
|
|
use std::time::Duration;
|
2019-11-26 23:54:46 +00:00
|
|
|
use types::*;
|
|
|
|
|
2020-08-17 08:05:13 +00:00
|
|
|
/// Defines how blocks should be replayed on states.
|
|
|
|
#[derive(PartialEq)]
|
|
|
|
pub enum BlockReplay {
|
|
|
|
/// Perform all transitions faithfully to the specification.
|
|
|
|
Accurate,
|
|
|
|
/// Don't compute state roots, eventually computing an invalid beacon state that can only be
|
|
|
|
/// used for obtaining shuffling.
|
|
|
|
InconsistentStateRoots,
|
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// On-disk database that stores finalized states efficiently.
|
|
|
|
///
|
|
|
|
/// Stores vector fields like the `block_roots` and `state_roots` separately, and only stores
|
|
|
|
/// intermittent "restore point" states pre-finalization.
|
2020-06-17 02:50:32 +00:00
|
|
|
#[derive(Debug)]
|
2020-06-16 01:34:04 +00:00
|
|
|
pub struct HotColdDB<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> {
|
2019-12-06 03:29:06 +00:00
|
|
|
/// The slot and state root at the point where the database is split between hot and cold.
|
2019-11-26 23:54:46 +00:00
|
|
|
///
|
2019-12-06 03:29:06 +00:00
|
|
|
/// States with slots less than `split.slot` are in the cold DB, while states with slots
|
|
|
|
/// greater than or equal are in the hot DB.
|
|
|
|
split: RwLock<Split>,
|
2020-02-10 00:30:21 +00:00
|
|
|
config: StoreConfig,
|
2019-11-26 23:54:46 +00:00
|
|
|
/// Cold database containing compact historical data.
|
2020-10-19 05:58:39 +00:00
|
|
|
pub cold_db: Cold,
|
2019-11-26 23:54:46 +00:00
|
|
|
/// Hot database containing duplicated but quick-to-access recent data.
|
2019-12-06 03:29:06 +00:00
|
|
|
///
|
|
|
|
/// The hot database also contains all blocks.
|
2020-10-19 05:58:39 +00:00
|
|
|
pub hot_db: Hot,
|
2020-02-10 00:30:21 +00:00
|
|
|
/// LRU cache of deserialized blocks. Updated whenever a block is loaded.
|
2020-02-10 23:19:36 +00:00
|
|
|
block_cache: Mutex<LruCache<Hash256, SignedBeaconBlock<E>>>,
|
2019-11-26 23:54:46 +00:00
|
|
|
/// Chain spec.
|
|
|
|
spec: ChainSpec,
|
|
|
|
/// Logger.
|
|
|
|
pub(crate) log: Logger,
|
2019-12-06 07:52:11 +00:00
|
|
|
/// Mere vessel for E.
|
|
|
|
_phantom: PhantomData<E>,
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
#[derive(Debug, PartialEq)]
|
2020-01-08 02:58:01 +00:00
|
|
|
pub enum HotColdDBError {
|
2020-09-30 02:36:07 +00:00
|
|
|
UnsupportedSchemaVersion {
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
target_version: SchemaVersion,
|
|
|
|
current_version: SchemaVersion,
|
2020-09-30 02:36:07 +00:00
|
|
|
},
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Recoverable error indicating that the database freeze point couldn't be updated
|
|
|
|
/// due to the finalized block not lying on an epoch boundary (should be infrequent).
|
|
|
|
FreezeSlotUnaligned(Slot),
|
2019-11-26 23:54:46 +00:00
|
|
|
FreezeSlotError {
|
|
|
|
current_split_slot: Slot,
|
|
|
|
proposed_split_slot: Slot,
|
|
|
|
},
|
2019-12-06 03:29:06 +00:00
|
|
|
MissingStateToFreeze(Hash256),
|
|
|
|
MissingRestorePointHash(u64),
|
|
|
|
MissingRestorePoint(Hash256),
|
2020-01-08 02:58:01 +00:00
|
|
|
MissingColdStateSummary(Hash256),
|
|
|
|
MissingHotStateSummary(Hash256),
|
|
|
|
MissingEpochBoundaryState(Hash256),
|
2019-12-06 03:29:06 +00:00
|
|
|
MissingSplitState(Hash256, Slot),
|
2020-01-08 02:58:01 +00:00
|
|
|
HotStateSummaryError(BeaconStateError),
|
2019-12-06 03:29:06 +00:00
|
|
|
RestorePointDecodeError(ssz::DecodeError),
|
|
|
|
BlockReplayBeaconError(BeaconStateError),
|
|
|
|
BlockReplaySlotError(SlotProcessingError),
|
|
|
|
BlockReplayBlockError(BlockProcessingError),
|
|
|
|
InvalidSlotsPerRestorePoint {
|
|
|
|
slots_per_restore_point: u64,
|
|
|
|
slots_per_historical_root: u64,
|
2020-01-09 10:05:56 +00:00
|
|
|
slots_per_epoch: u64,
|
2019-12-06 03:29:06 +00:00
|
|
|
},
|
|
|
|
RestorePointBlockHashError(BeaconStateError),
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
IterationError {
|
|
|
|
unexpected_key: BytesKey,
|
|
|
|
},
|
2021-06-01 06:59:43 +00:00
|
|
|
AttestationStateIsFinalized {
|
|
|
|
split_slot: Slot,
|
|
|
|
request_slot: Option<Slot>,
|
|
|
|
state_root: Hash256,
|
|
|
|
},
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
2020-07-01 02:45:57 +00:00
|
|
|
impl<E: EthSpec> HotColdDB<E, MemoryStore<E>, MemoryStore<E>> {
|
|
|
|
pub fn open_ephemeral(
|
|
|
|
config: StoreConfig,
|
|
|
|
spec: ChainSpec,
|
|
|
|
log: Logger,
|
|
|
|
) -> Result<HotColdDB<E, MemoryStore<E>, MemoryStore<E>>, Error> {
|
|
|
|
Self::verify_slots_per_restore_point(config.slots_per_restore_point)?;
|
|
|
|
|
|
|
|
let db = HotColdDB {
|
|
|
|
split: RwLock::new(Split::default()),
|
|
|
|
cold_db: MemoryStore::open(),
|
|
|
|
hot_db: MemoryStore::open(),
|
|
|
|
block_cache: Mutex::new(LruCache::new(config.block_cache_size)),
|
|
|
|
config,
|
|
|
|
spec,
|
|
|
|
log,
|
|
|
|
_phantom: PhantomData,
|
|
|
|
};
|
|
|
|
|
|
|
|
Ok(db)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl<E: EthSpec> HotColdDB<E, LevelDB<E>, LevelDB<E>> {
|
|
|
|
/// Open a new or existing database, with the given paths to the hot and cold DBs.
|
|
|
|
///
|
|
|
|
/// The `slots_per_restore_point` parameter must be a divisor of `SLOTS_PER_HISTORICAL_ROOT`.
|
2021-03-04 01:25:12 +00:00
|
|
|
///
|
|
|
|
/// The `migrate_schema` function is passed in so that the parent `BeaconChain` can provide
|
|
|
|
/// context and access `BeaconChain`-level code without creating a circular dependency.
|
2020-07-01 02:45:57 +00:00
|
|
|
pub fn open(
|
|
|
|
hot_path: &Path,
|
|
|
|
cold_path: &Path,
|
2021-03-04 01:25:12 +00:00
|
|
|
migrate_schema: impl FnOnce(Arc<Self>, SchemaVersion, SchemaVersion) -> Result<(), Error>,
|
2020-07-01 02:45:57 +00:00
|
|
|
config: StoreConfig,
|
|
|
|
spec: ChainSpec,
|
|
|
|
log: Logger,
|
2021-03-04 01:25:12 +00:00
|
|
|
) -> Result<Arc<Self>, Error> {
|
2020-07-01 02:45:57 +00:00
|
|
|
Self::verify_slots_per_restore_point(config.slots_per_restore_point)?;
|
|
|
|
|
2021-03-04 01:25:12 +00:00
|
|
|
let db = Arc::new(HotColdDB {
|
2020-07-01 02:45:57 +00:00
|
|
|
split: RwLock::new(Split::default()),
|
|
|
|
cold_db: LevelDB::open(cold_path)?,
|
|
|
|
hot_db: LevelDB::open(hot_path)?,
|
|
|
|
block_cache: Mutex::new(LruCache::new(config.block_cache_size)),
|
|
|
|
config,
|
|
|
|
spec,
|
|
|
|
log,
|
|
|
|
_phantom: PhantomData,
|
2021-03-04 01:25:12 +00:00
|
|
|
});
|
2020-07-01 02:45:57 +00:00
|
|
|
|
2020-09-30 02:36:07 +00:00
|
|
|
// Ensure that the schema version of the on-disk database matches the software.
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
// If the version is mismatched, an automatic migration will be attempted.
|
2020-09-30 02:36:07 +00:00
|
|
|
if let Some(schema_version) = db.load_schema_version()? {
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
debug!(
|
|
|
|
db.log,
|
|
|
|
"Attempting schema migration";
|
|
|
|
"from_version" => schema_version.as_u64(),
|
|
|
|
"to_version" => CURRENT_SCHEMA_VERSION.as_u64(),
|
|
|
|
);
|
2021-03-04 01:25:12 +00:00
|
|
|
migrate_schema(db.clone(), schema_version, CURRENT_SCHEMA_VERSION)?;
|
2020-09-30 02:36:07 +00:00
|
|
|
} else {
|
|
|
|
db.store_schema_version(CURRENT_SCHEMA_VERSION)?;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Ensure that any on-disk config is compatible with the supplied config.
|
|
|
|
if let Some(disk_config) = db.load_config()? {
|
|
|
|
db.config.check_compatibility(&disk_config)?;
|
|
|
|
}
|
|
|
|
db.store_config()?;
|
|
|
|
|
2020-07-01 02:45:57 +00:00
|
|
|
// Load the previous split slot from the database (if any). This ensures we can
|
|
|
|
// stop and restart correctly.
|
|
|
|
if let Some(split) = db.load_split()? {
|
2020-08-26 00:01:06 +00:00
|
|
|
info!(
|
|
|
|
db.log,
|
|
|
|
"Hot-Cold DB initialized";
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
"version" => CURRENT_SCHEMA_VERSION.as_u64(),
|
2020-08-26 00:01:06 +00:00
|
|
|
"split_slot" => split.slot,
|
|
|
|
"split_state" => format!("{:?}", split.state_root)
|
|
|
|
);
|
2020-07-01 02:45:57 +00:00
|
|
|
*db.split.write() = split;
|
|
|
|
}
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
|
2020-11-17 09:10:53 +00:00
|
|
|
// Run a garbage collection pass.
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
db.remove_garbage()?;
|
|
|
|
|
2020-11-17 09:10:53 +00:00
|
|
|
// If configured, run a foreground compaction pass.
|
|
|
|
if db.config.compact_on_init {
|
|
|
|
info!(db.log, "Running foreground compaction");
|
|
|
|
db.compact()?;
|
|
|
|
info!(db.log, "Foreground compaction complete");
|
|
|
|
}
|
|
|
|
|
2020-07-01 02:45:57 +00:00
|
|
|
Ok(db)
|
|
|
|
}
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
|
|
|
|
/// Return an iterator over the state roots of all temporary states.
|
2021-01-19 00:34:28 +00:00
|
|
|
pub fn iter_temporary_state_roots(&self) -> impl Iterator<Item = Result<Hash256, Error>> + '_ {
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
let column = DBColumn::BeaconStateTemporary;
|
|
|
|
let start_key =
|
|
|
|
BytesKey::from_vec(get_key_for_col(column.into(), Hash256::zero().as_bytes()));
|
|
|
|
|
|
|
|
let keys_iter = self.hot_db.keys_iter();
|
|
|
|
keys_iter.seek(&start_key);
|
|
|
|
|
|
|
|
keys_iter
|
|
|
|
.take_while(move |key| key.matches_column(column))
|
|
|
|
.map(move |bytes_key| {
|
|
|
|
bytes_key.remove_column(column).ok_or_else(|| {
|
|
|
|
HotColdDBError::IterationError {
|
|
|
|
unexpected_key: bytes_key,
|
|
|
|
}
|
|
|
|
.into()
|
|
|
|
})
|
|
|
|
})
|
|
|
|
}
|
2020-07-01 02:45:57 +00:00
|
|
|
}
|
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> HotColdDB<E, Hot, Cold> {
|
2020-02-10 00:30:21 +00:00
|
|
|
/// Store a block and update the LRU cache.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn put_block(
|
|
|
|
&self,
|
|
|
|
block_root: &Hash256,
|
|
|
|
block: SignedBeaconBlock<E>,
|
|
|
|
) -> Result<(), Error> {
|
2020-02-10 00:30:21 +00:00
|
|
|
// Store on disk.
|
2021-07-09 06:15:32 +00:00
|
|
|
self.hot_db
|
|
|
|
.do_atomically(vec![beacon_block_as_kv_store_op(block_root, &block)])?;
|
2020-02-10 00:30:21 +00:00
|
|
|
|
|
|
|
// Update cache.
|
|
|
|
self.block_cache.lock().put(*block_root, block);
|
|
|
|
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Fetch a block from the store.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn get_block(&self, block_root: &Hash256) -> Result<Option<SignedBeaconBlock<E>>, Error> {
|
2020-02-10 00:30:21 +00:00
|
|
|
metrics::inc_counter(&metrics::BEACON_BLOCK_GET_COUNT);
|
|
|
|
|
|
|
|
// Check the cache.
|
|
|
|
if let Some(block) = self.block_cache.lock().get(block_root) {
|
|
|
|
metrics::inc_counter(&metrics::BEACON_BLOCK_CACHE_HIT_COUNT);
|
|
|
|
return Ok(Some(block.clone()));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Fetch from database.
|
2021-07-09 06:15:32 +00:00
|
|
|
match self
|
|
|
|
.hot_db
|
|
|
|
.get_bytes(DBColumn::BeaconBlock.into(), block_root.as_bytes())?
|
|
|
|
{
|
|
|
|
Some(block_bytes) => {
|
|
|
|
// Deserialize.
|
|
|
|
let block = SignedBeaconBlock::from_ssz_bytes(&block_bytes, &self.spec)?;
|
|
|
|
|
2020-02-10 00:30:21 +00:00
|
|
|
// Add to cache.
|
|
|
|
self.block_cache.lock().put(*block_root, block.clone());
|
2021-07-09 06:15:32 +00:00
|
|
|
|
2020-02-10 00:30:21 +00:00
|
|
|
Ok(Some(block))
|
|
|
|
}
|
|
|
|
None => Ok(None),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-07-09 06:15:32 +00:00
|
|
|
/// Determine whether a block exists in the database.
|
|
|
|
pub fn block_exists(&self, block_root: &Hash256) -> Result<bool, Error> {
|
|
|
|
self.hot_db
|
|
|
|
.key_exists(DBColumn::BeaconBlock.into(), block_root.as_bytes())
|
|
|
|
}
|
|
|
|
|
2020-03-04 05:48:35 +00:00
|
|
|
/// Delete a block from the store and the block cache.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn delete_block(&self, block_root: &Hash256) -> Result<(), Error> {
|
2020-03-04 05:48:35 +00:00
|
|
|
self.block_cache.lock().pop(block_root);
|
2021-07-09 06:15:32 +00:00
|
|
|
self.hot_db
|
|
|
|
.key_delete(DBColumn::BeaconBlock.into(), block_root.as_bytes())
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn put_state_summary(
|
2020-05-31 22:13:49 +00:00
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
summary: HotStateSummary,
|
|
|
|
) -> Result<(), Error> {
|
|
|
|
self.hot_db.put(state_root, &summary).map_err(Into::into)
|
2020-03-04 05:48:35 +00:00
|
|
|
}
|
|
|
|
|
2019-11-26 23:54:46 +00:00
|
|
|
/// Store a state in the store.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn put_state(&self, state_root: &Hash256, state: &BeaconState<E>) -> Result<(), Error> {
|
2021-06-18 05:58:01 +00:00
|
|
|
let mut ops: Vec<KeyValueStoreOp> = Vec::new();
|
2021-07-09 06:15:32 +00:00
|
|
|
if state.slot() < self.get_split_slot() {
|
2020-07-01 02:45:57 +00:00
|
|
|
self.store_cold_state(state_root, &state, &mut ops)?;
|
|
|
|
self.cold_db.do_atomically(ops)
|
2019-11-26 23:54:46 +00:00
|
|
|
} else {
|
2020-07-01 02:45:57 +00:00
|
|
|
self.store_hot_state(state_root, state, &mut ops)?;
|
|
|
|
self.hot_db.do_atomically(ops)
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Fetch a state from the store.
|
2020-08-12 07:00:00 +00:00
|
|
|
///
|
|
|
|
/// If `slot` is provided then it will be used as a hint as to which database should
|
|
|
|
/// be checked. Importantly, if the slot hint is provided and indicates a slot that lies
|
|
|
|
/// in the freezer database, then only the freezer database will be accessed and `Ok(None)`
|
|
|
|
/// will be returned if the provided `state_root` doesn't match the state root of the
|
|
|
|
/// frozen state at `slot`. Consequently, if a state from a non-canonical chain is desired, it's
|
|
|
|
/// best to set `slot` to `None`, or call `load_hot_state` directly.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn get_state(
|
2019-11-26 23:54:46 +00:00
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
slot: Option<Slot>,
|
2020-02-10 00:30:21 +00:00
|
|
|
) -> Result<Option<BeaconState<E>>, Error> {
|
|
|
|
metrics::inc_counter(&metrics::BEACON_STATE_GET_COUNT);
|
|
|
|
|
2019-11-26 23:54:46 +00:00
|
|
|
if let Some(slot) = slot {
|
|
|
|
if slot < self.get_split_slot() {
|
2020-08-12 07:00:00 +00:00
|
|
|
// Although we could avoid a DB lookup by shooting straight for the
|
|
|
|
// frozen state using `load_cold_state_by_slot`, that would be incorrect
|
|
|
|
// in the case where the caller provides a `state_root` that's off the canonical
|
|
|
|
// chain. This way we avoid returning a state that doesn't match `state_root`.
|
|
|
|
self.load_cold_state(state_root)
|
2019-11-26 23:54:46 +00:00
|
|
|
} else {
|
2020-08-17 08:05:13 +00:00
|
|
|
self.load_hot_state(state_root, BlockReplay::Accurate)
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
} else {
|
2020-08-17 08:05:13 +00:00
|
|
|
match self.load_hot_state(state_root, BlockReplay::Accurate)? {
|
2019-11-26 23:54:46 +00:00
|
|
|
Some(state) => Ok(Some(state)),
|
2020-01-08 02:58:01 +00:00
|
|
|
None => self.load_cold_state(state_root),
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-08-17 08:05:13 +00:00
|
|
|
/// Fetch a state from the store, but don't compute all of the values when replaying blocks
|
|
|
|
/// upon that state (e.g., state roots). Additionally, only states from the hot store are
|
|
|
|
/// returned.
|
|
|
|
///
|
|
|
|
/// See `Self::get_state` for information about `slot`.
|
|
|
|
///
|
|
|
|
/// ## Warning
|
|
|
|
///
|
|
|
|
/// The returned state **is not a valid beacon state**, it can only be used for obtaining
|
2021-03-17 05:09:57 +00:00
|
|
|
/// shuffling to process attestations. At least the following components of the state will be
|
|
|
|
/// broken/invalid:
|
|
|
|
///
|
|
|
|
/// - `state.state_roots`
|
|
|
|
/// - `state.block_roots`
|
2020-08-17 08:05:13 +00:00
|
|
|
pub fn get_inconsistent_state_for_attestation_verification_only(
|
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
slot: Option<Slot>,
|
|
|
|
) -> Result<Option<BeaconState<E>>, Error> {
|
|
|
|
metrics::inc_counter(&metrics::BEACON_STATE_GET_COUNT);
|
|
|
|
|
2021-06-01 06:59:43 +00:00
|
|
|
let split_slot = self.get_split_slot();
|
|
|
|
|
|
|
|
if slot.map_or(false, |slot| slot < split_slot) {
|
|
|
|
Err(HotColdDBError::AttestationStateIsFinalized {
|
|
|
|
split_slot,
|
|
|
|
request_slot: slot,
|
|
|
|
state_root: *state_root,
|
|
|
|
}
|
|
|
|
.into())
|
2020-08-17 08:05:13 +00:00
|
|
|
} else {
|
|
|
|
self.load_hot_state(state_root, BlockReplay::InconsistentStateRoots)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-03-04 05:48:35 +00:00
|
|
|
/// Delete a state, ensuring it is removed from the LRU cache, as well as from on-disk.
|
|
|
|
///
|
|
|
|
/// It is assumed that all states being deleted reside in the hot DB, even if their slot is less
|
|
|
|
/// than the split point. You shouldn't delete states from the finalized portion of the chain
|
|
|
|
/// (which are frozen, and won't be deleted), or valid descendents of the finalized checkpoint
|
|
|
|
/// (which will be deleted by this function but shouldn't be).
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn delete_state(&self, state_root: &Hash256, slot: Slot) -> Result<(), Error> {
|
2020-03-04 05:48:35 +00:00
|
|
|
// Delete the state summary.
|
|
|
|
self.hot_db
|
|
|
|
.key_delete(DBColumn::BeaconStateSummary.into(), state_root.as_bytes())?;
|
|
|
|
|
|
|
|
// Delete the full state if it lies on an epoch boundary.
|
|
|
|
if slot % E::slots_per_epoch() == 0 {
|
|
|
|
self.hot_db
|
|
|
|
.key_delete(DBColumn::BeaconState.into(), state_root.as_bytes())?;
|
|
|
|
}
|
|
|
|
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn forwards_block_roots_iterator(
|
2019-12-06 07:52:11 +00:00
|
|
|
store: Arc<Self>,
|
|
|
|
start_slot: Slot,
|
|
|
|
end_state: BeaconState<E>,
|
|
|
|
end_block_root: Hash256,
|
|
|
|
spec: &ChainSpec,
|
2020-06-16 01:34:04 +00:00
|
|
|
) -> Result<impl Iterator<Item = Result<(Hash256, Slot), Error>>, Error> {
|
2019-12-06 07:52:11 +00:00
|
|
|
HybridForwardsBlockRootsIterator::new(store, start_slot, end_state, end_block_root, spec)
|
|
|
|
}
|
2020-01-08 02:58:01 +00:00
|
|
|
|
2021-07-06 02:38:53 +00:00
|
|
|
pub fn forwards_state_roots_iterator(
|
|
|
|
store: Arc<Self>,
|
|
|
|
start_slot: Slot,
|
|
|
|
end_state_root: Hash256,
|
|
|
|
end_state: BeaconState<E>,
|
|
|
|
spec: &ChainSpec,
|
|
|
|
) -> Result<impl Iterator<Item = Result<(Hash256, Slot), Error>>, Error> {
|
|
|
|
HybridForwardsStateRootsIterator::new(store, start_slot, end_state, end_state_root, spec)
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Load an epoch boundary state by using the hot state summary look-up.
|
|
|
|
///
|
|
|
|
/// Will fall back to the cold DB if a hot state summary is not found.
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn load_epoch_boundary_state(
|
2020-01-08 02:58:01 +00:00
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
) -> Result<Option<BeaconState<E>>, Error> {
|
|
|
|
if let Some(HotStateSummary {
|
|
|
|
epoch_boundary_state_root,
|
|
|
|
..
|
|
|
|
}) = self.load_hot_state_summary(state_root)?
|
|
|
|
{
|
2020-02-10 00:30:21 +00:00
|
|
|
// NOTE: minor inefficiency here because we load an unnecessary hot state summary
|
2020-08-17 08:05:13 +00:00
|
|
|
//
|
|
|
|
// `BlockReplay` should be irrelevant here since we never replay blocks for an epoch
|
|
|
|
// boundary state in the hot DB.
|
2020-01-08 02:58:01 +00:00
|
|
|
let state = self
|
2020-08-17 08:05:13 +00:00
|
|
|
.load_hot_state(&epoch_boundary_state_root, BlockReplay::Accurate)?
|
2020-12-03 01:10:26 +00:00
|
|
|
.ok_or(HotColdDBError::MissingEpochBoundaryState(
|
|
|
|
epoch_boundary_state_root,
|
|
|
|
))?;
|
2020-01-08 02:58:01 +00:00
|
|
|
Ok(Some(state))
|
|
|
|
} else {
|
|
|
|
// Try the cold DB
|
|
|
|
match self.load_cold_state_slot(state_root)? {
|
|
|
|
Some(state_slot) => {
|
|
|
|
let epoch_boundary_slot =
|
|
|
|
state_slot / E::slots_per_epoch() * E::slots_per_epoch();
|
|
|
|
self.load_cold_state_by_slot(epoch_boundary_slot).map(Some)
|
|
|
|
}
|
|
|
|
None => Ok(None),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2020-05-31 22:13:49 +00:00
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn put_item<I: StoreItem>(&self, key: &Hash256, item: &I) -> Result<(), Error> {
|
2020-05-31 22:13:49 +00:00
|
|
|
self.hot_db.put(key, item)
|
|
|
|
}
|
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn get_item<I: StoreItem>(&self, key: &Hash256) -> Result<Option<I>, Error> {
|
2020-05-31 22:13:49 +00:00
|
|
|
self.hot_db.get(key)
|
|
|
|
}
|
|
|
|
|
2020-06-16 01:34:04 +00:00
|
|
|
pub fn item_exists<I: StoreItem>(&self, key: &Hash256) -> Result<bool, Error> {
|
2020-05-31 22:13:49 +00:00
|
|
|
self.hot_db.exists::<I>(key)
|
|
|
|
}
|
|
|
|
|
2020-10-19 05:58:39 +00:00
|
|
|
/// Convert a batch of `StoreOp` to a batch of `KeyValueStoreOp`.
|
|
|
|
pub fn convert_to_kv_batch(&self, batch: &[StoreOp<E>]) -> Result<Vec<KeyValueStoreOp>, Error> {
|
|
|
|
let mut key_value_batch = Vec::with_capacity(batch.len());
|
|
|
|
for op in batch {
|
2020-06-16 01:34:04 +00:00
|
|
|
match op {
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::PutBlock(block_root, block) => {
|
2021-07-09 06:15:32 +00:00
|
|
|
key_value_batch.push(beacon_block_as_kv_store_op(block_root, block));
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
StoreOp::PutState(state_root, state) => {
|
|
|
|
self.store_hot_state(state_root, state, &mut key_value_batch)?;
|
2020-07-01 02:45:57 +00:00
|
|
|
}
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::PutStateSummary(state_root, summary) => {
|
|
|
|
key_value_batch.push(summary.as_kv_store_op(*state_root));
|
2020-07-01 02:45:57 +00:00
|
|
|
}
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::PutStateTemporaryFlag(state_root) => {
|
|
|
|
key_value_batch.push(TemporaryFlag.as_kv_store_op(*state_root));
|
2020-07-01 02:45:57 +00:00
|
|
|
}
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::DeleteStateTemporaryFlag(state_root) => {
|
|
|
|
let db_key =
|
|
|
|
get_key_for_col(TemporaryFlag::db_column().into(), state_root.as_bytes());
|
|
|
|
key_value_batch.push(KeyValueStoreOp::DeleteKey(db_key));
|
|
|
|
}
|
|
|
|
|
|
|
|
StoreOp::DeleteBlock(block_root) => {
|
|
|
|
let key = get_key_for_col(DBColumn::BeaconBlock.into(), block_root.as_bytes());
|
2020-06-16 01:34:04 +00:00
|
|
|
key_value_batch.push(KeyValueStoreOp::DeleteKey(key));
|
|
|
|
}
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::DeleteState(state_root, slot) => {
|
|
|
|
let state_summary_key =
|
|
|
|
get_key_for_col(DBColumn::BeaconStateSummary.into(), state_root.as_bytes());
|
2020-06-16 01:34:04 +00:00
|
|
|
key_value_batch.push(KeyValueStoreOp::DeleteKey(state_summary_key));
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
if slot.map_or(true, |slot| slot % E::slots_per_epoch() == 0) {
|
2020-06-16 01:34:04 +00:00
|
|
|
let state_key =
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
get_key_for_col(DBColumn::BeaconState.into(), state_root.as_bytes());
|
2020-06-16 01:34:04 +00:00
|
|
|
key_value_batch.push(KeyValueStoreOp::DeleteKey(state_key));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2020-10-19 05:58:39 +00:00
|
|
|
Ok(key_value_batch)
|
|
|
|
}
|
|
|
|
|
|
|
|
pub fn do_atomically(&self, batch: Vec<StoreOp<E>>) -> Result<(), Error> {
|
|
|
|
let mut guard = self.block_cache.lock();
|
|
|
|
|
|
|
|
self.hot_db
|
|
|
|
.do_atomically(self.convert_to_kv_batch(&batch)?)?;
|
2020-06-16 01:34:04 +00:00
|
|
|
|
2020-07-01 02:45:57 +00:00
|
|
|
for op in &batch {
|
2020-05-31 22:13:49 +00:00
|
|
|
match op {
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::PutBlock(block_root, block) => {
|
|
|
|
guard.put(*block_root, (**block).clone());
|
2020-07-01 02:45:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
StoreOp::PutState(_, _) => (),
|
|
|
|
|
|
|
|
StoreOp::PutStateSummary(_, _) => (),
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
StoreOp::PutStateTemporaryFlag(_) => (),
|
|
|
|
|
|
|
|
StoreOp::DeleteStateTemporaryFlag(_) => (),
|
|
|
|
|
|
|
|
StoreOp::DeleteBlock(block_root) => {
|
|
|
|
guard.pop(block_root);
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
2020-07-01 02:45:57 +00:00
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
StoreOp::DeleteState(_, _) => (),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
Ok(())
|
|
|
|
}
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Store a post-finalization state efficiently in the hot database.
|
|
|
|
///
|
|
|
|
/// On an epoch boundary, store a full state. On an intermediate slot, store
|
|
|
|
/// just a backpointer to the nearest epoch boundary.
|
|
|
|
pub fn store_hot_state(
|
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
2020-04-06 00:53:33 +00:00
|
|
|
state: &BeaconState<E>,
|
2020-07-01 02:45:57 +00:00
|
|
|
ops: &mut Vec<KeyValueStoreOp>,
|
2020-01-08 02:58:01 +00:00
|
|
|
) -> Result<(), Error> {
|
|
|
|
// On the epoch boundary, store the full state.
|
2021-07-09 06:15:32 +00:00
|
|
|
if state.slot() % E::slots_per_epoch() == 0 {
|
2020-01-08 02:58:01 +00:00
|
|
|
trace!(
|
|
|
|
self.log,
|
|
|
|
"Storing full state on epoch boundary";
|
2021-07-09 06:15:32 +00:00
|
|
|
"slot" => state.slot().as_u64(),
|
2020-01-08 02:58:01 +00:00
|
|
|
"state_root" => format!("{:?}", state_root)
|
|
|
|
);
|
2020-07-01 02:45:57 +00:00
|
|
|
store_full_state(state_root, &state, ops)?;
|
2020-01-08 02:58:01 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Store a summary of the state.
|
|
|
|
// We store one even for the epoch boundary states, as we may need their slots
|
|
|
|
// when doing a look up by state root.
|
2020-07-01 02:45:57 +00:00
|
|
|
let hot_state_summary = HotStateSummary::new(state_root, state)?;
|
|
|
|
let op = hot_state_summary.as_kv_store_op(*state_root);
|
|
|
|
ops.push(op);
|
2020-01-08 02:58:01 +00:00
|
|
|
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Load a post-finalization state from the hot database.
|
|
|
|
///
|
|
|
|
/// Will replay blocks from the nearest epoch boundary.
|
2020-08-17 08:05:13 +00:00
|
|
|
pub fn load_hot_state(
|
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
block_replay: BlockReplay,
|
|
|
|
) -> Result<Option<BeaconState<E>>, Error> {
|
2020-02-10 00:30:21 +00:00
|
|
|
metrics::inc_counter(&metrics::BEACON_STATE_HOT_GET_COUNT);
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
// If the state is marked as temporary, do not return it. It will become visible
|
|
|
|
// only once its transaction commits and deletes its temporary flag.
|
|
|
|
if self.load_state_temporary_flag(state_root)?.is_some() {
|
|
|
|
return Ok(None);
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
if let Some(HotStateSummary {
|
|
|
|
slot,
|
|
|
|
latest_block_root,
|
|
|
|
epoch_boundary_state_root,
|
|
|
|
}) = self.load_hot_state_summary(state_root)?
|
|
|
|
{
|
2021-07-09 06:15:32 +00:00
|
|
|
let boundary_state =
|
|
|
|
get_full_state(&self.hot_db, &epoch_boundary_state_root, &self.spec)?.ok_or(
|
|
|
|
HotColdDBError::MissingEpochBoundaryState(epoch_boundary_state_root),
|
|
|
|
)?;
|
2020-01-08 02:58:01 +00:00
|
|
|
|
|
|
|
// Optimization to avoid even *thinking* about replaying blocks if we're already
|
|
|
|
// on an epoch boundary.
|
2020-02-10 00:30:21 +00:00
|
|
|
let state = if slot % E::slots_per_epoch() == 0 {
|
|
|
|
boundary_state
|
2020-01-08 02:58:01 +00:00
|
|
|
} else {
|
2020-02-10 00:30:21 +00:00
|
|
|
let blocks =
|
2021-07-09 06:15:32 +00:00
|
|
|
self.load_blocks_to_replay(boundary_state.slot(), slot, latest_block_root)?;
|
2020-08-17 08:05:13 +00:00
|
|
|
self.replay_blocks(boundary_state, blocks, slot, block_replay)?
|
2020-02-10 00:30:21 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
Ok(Some(state))
|
2020-01-08 02:58:01 +00:00
|
|
|
} else {
|
|
|
|
Ok(None)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Store a pre-finalization state in the freezer database.
|
|
|
|
///
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Will log a warning and not store anything if the state does not lie on a restore point
|
|
|
|
/// boundary.
|
|
|
|
pub fn store_cold_state(
|
2019-11-26 23:54:46 +00:00
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
state: &BeaconState<E>,
|
2020-07-01 02:45:57 +00:00
|
|
|
ops: &mut Vec<KeyValueStoreOp>,
|
2019-11-26 23:54:46 +00:00
|
|
|
) -> Result<(), Error> {
|
2021-07-09 06:15:32 +00:00
|
|
|
if state.slot() % self.config.slots_per_restore_point != 0 {
|
2019-12-06 03:29:06 +00:00
|
|
|
warn!(
|
|
|
|
self.log,
|
|
|
|
"Not storing non-restore point state in freezer";
|
2021-07-09 06:15:32 +00:00
|
|
|
"slot" => state.slot().as_u64(),
|
2019-12-06 03:29:06 +00:00
|
|
|
"state_root" => format!("{:?}", state_root)
|
|
|
|
);
|
|
|
|
return Ok(());
|
|
|
|
}
|
|
|
|
|
2019-11-26 23:54:46 +00:00
|
|
|
trace!(
|
|
|
|
self.log,
|
2019-12-06 03:29:06 +00:00
|
|
|
"Creating restore point";
|
2021-07-09 06:15:32 +00:00
|
|
|
"slot" => state.slot(),
|
2019-11-26 23:54:46 +00:00
|
|
|
"state_root" => format!("{:?}", state_root)
|
|
|
|
);
|
2019-12-06 03:29:06 +00:00
|
|
|
|
2019-11-26 23:54:46 +00:00
|
|
|
// 1. Convert to PartialBeaconState and store that in the DB.
|
|
|
|
let partial_state = PartialBeaconState::from_state_forgetful(state);
|
2020-07-01 02:45:57 +00:00
|
|
|
let op = partial_state.as_kv_store_op(*state_root);
|
|
|
|
ops.push(op);
|
2019-11-26 23:54:46 +00:00
|
|
|
|
|
|
|
// 2. Store updated vector entries.
|
|
|
|
let db = &self.cold_db;
|
2020-07-01 02:45:57 +00:00
|
|
|
store_updated_vector(BlockRoots, db, state, &self.spec, ops)?;
|
|
|
|
store_updated_vector(StateRoots, db, state, &self.spec, ops)?;
|
|
|
|
store_updated_vector(HistoricalRoots, db, state, &self.spec, ops)?;
|
|
|
|
store_updated_vector(RandaoMixes, db, state, &self.spec, ops)?;
|
2019-11-26 23:54:46 +00:00
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
// 3. Store restore point.
|
2021-07-09 06:15:32 +00:00
|
|
|
let restore_point_index = state.slot().as_u64() / self.config.slots_per_restore_point;
|
2020-07-01 02:45:57 +00:00
|
|
|
self.store_restore_point_hash(restore_point_index, *state_root, ops);
|
2019-12-06 03:29:06 +00:00
|
|
|
|
2019-11-26 23:54:46 +00:00
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Try to load a pre-finalization state from the freezer database.
|
|
|
|
///
|
|
|
|
/// Return `None` if no state with `state_root` lies in the freezer.
|
|
|
|
pub fn load_cold_state(&self, state_root: &Hash256) -> Result<Option<BeaconState<E>>, Error> {
|
|
|
|
match self.load_cold_state_slot(state_root)? {
|
|
|
|
Some(slot) => self.load_cold_state_by_slot(slot).map(Some),
|
|
|
|
None => Ok(None),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Load a pre-finalization state from the freezer database.
|
|
|
|
///
|
|
|
|
/// Will reconstruct the state if it lies between restore points.
|
2020-01-08 02:58:01 +00:00
|
|
|
pub fn load_cold_state_by_slot(&self, slot: Slot) -> Result<BeaconState<E>, Error> {
|
2020-02-10 00:30:21 +00:00
|
|
|
if slot % self.config.slots_per_restore_point == 0 {
|
|
|
|
let restore_point_idx = slot.as_u64() / self.config.slots_per_restore_point;
|
2020-01-08 02:58:01 +00:00
|
|
|
self.load_restore_point_by_index(restore_point_idx)
|
2019-12-06 03:29:06 +00:00
|
|
|
} else {
|
2020-01-08 02:58:01 +00:00
|
|
|
self.load_cold_intermediate_state(slot)
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Load a restore point state by its `state_root`.
|
2019-12-06 07:52:11 +00:00
|
|
|
fn load_restore_point(&self, state_root: &Hash256) -> Result<BeaconState<E>, Error> {
|
2021-07-09 06:15:32 +00:00
|
|
|
let partial_state_bytes = self
|
2020-05-25 00:26:54 +00:00
|
|
|
.cold_db
|
2021-07-09 06:15:32 +00:00
|
|
|
.get_bytes(DBColumn::BeaconState.into(), state_root.as_bytes())?
|
2020-01-08 02:58:01 +00:00
|
|
|
.ok_or_else(|| HotColdDBError::MissingRestorePoint(*state_root))?;
|
2021-07-09 06:15:32 +00:00
|
|
|
let mut partial_state: PartialBeaconState<E> =
|
|
|
|
PartialBeaconState::from_ssz_bytes(&partial_state_bytes, &self.spec)?;
|
2019-11-26 23:54:46 +00:00
|
|
|
|
|
|
|
// Fill in the fields of the partial state.
|
|
|
|
partial_state.load_block_roots(&self.cold_db, &self.spec)?;
|
|
|
|
partial_state.load_state_roots(&self.cold_db, &self.spec)?;
|
|
|
|
partial_state.load_historical_roots(&self.cold_db, &self.spec)?;
|
|
|
|
partial_state.load_randao_mixes(&self.cold_db, &self.spec)?;
|
|
|
|
|
2021-03-26 04:53:57 +00:00
|
|
|
partial_state.try_into()
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/// Load a restore point state by its `restore_point_index`.
|
2019-12-06 07:52:11 +00:00
|
|
|
fn load_restore_point_by_index(
|
2019-12-06 03:29:06 +00:00
|
|
|
&self,
|
|
|
|
restore_point_index: u64,
|
|
|
|
) -> Result<BeaconState<E>, Error> {
|
|
|
|
let state_root = self.load_restore_point_hash(restore_point_index)?;
|
|
|
|
self.load_restore_point(&state_root)
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Load a frozen state that lies between restore points.
|
|
|
|
fn load_cold_intermediate_state(&self, slot: Slot) -> Result<BeaconState<E>, Error> {
|
2019-12-06 03:29:06 +00:00
|
|
|
// 1. Load the restore points either side of the intermediate state.
|
2020-02-10 00:30:21 +00:00
|
|
|
let low_restore_point_idx = slot.as_u64() / self.config.slots_per_restore_point;
|
2019-12-06 03:29:06 +00:00
|
|
|
let high_restore_point_idx = low_restore_point_idx + 1;
|
|
|
|
|
|
|
|
// Acquire the read lock, so that the split can't change while this is happening.
|
|
|
|
let split = self.split.read();
|
|
|
|
|
|
|
|
let low_restore_point = self.load_restore_point_by_index(low_restore_point_idx)?;
|
|
|
|
// If the slot of the high point lies outside the freezer, use the split state
|
|
|
|
// as the upper restore point.
|
2020-02-10 00:30:21 +00:00
|
|
|
let high_restore_point = if high_restore_point_idx * self.config.slots_per_restore_point
|
2019-12-06 03:29:06 +00:00
|
|
|
>= split.slot.as_u64()
|
|
|
|
{
|
2020-12-03 01:10:26 +00:00
|
|
|
self.get_state(&split.state_root, Some(split.slot))?.ok_or(
|
|
|
|
HotColdDBError::MissingSplitState(split.state_root, split.slot),
|
|
|
|
)?
|
2019-12-06 03:29:06 +00:00
|
|
|
} else {
|
|
|
|
self.load_restore_point_by_index(high_restore_point_idx)?
|
|
|
|
};
|
|
|
|
|
|
|
|
// 2. Load the blocks from the high restore point back to the low restore point.
|
|
|
|
let blocks = self.load_blocks_to_replay(
|
2021-07-09 06:15:32 +00:00
|
|
|
low_restore_point.slot(),
|
2019-12-06 03:29:06 +00:00
|
|
|
slot,
|
|
|
|
self.get_high_restore_point_block_root(&high_restore_point, slot)?,
|
|
|
|
)?;
|
|
|
|
|
|
|
|
// 3. Replay the blocks on top of the low restore point.
|
2020-08-17 08:05:13 +00:00
|
|
|
self.replay_blocks(low_restore_point, blocks, slot, BlockReplay::Accurate)
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/// Get a suitable block root for backtracking from `high_restore_point` to the state at `slot`.
|
|
|
|
///
|
|
|
|
/// Defaults to the block root for `slot`, which *should* be in range.
|
2019-12-06 07:52:11 +00:00
|
|
|
fn get_high_restore_point_block_root(
|
2019-12-06 03:29:06 +00:00
|
|
|
&self,
|
|
|
|
high_restore_point: &BeaconState<E>,
|
|
|
|
slot: Slot,
|
2020-01-08 02:58:01 +00:00
|
|
|
) -> Result<Hash256, HotColdDBError> {
|
2019-12-06 03:29:06 +00:00
|
|
|
high_restore_point
|
|
|
|
.get_block_root(slot)
|
|
|
|
.or_else(|_| high_restore_point.get_oldest_block_root())
|
|
|
|
.map(|x| *x)
|
2020-01-08 02:58:01 +00:00
|
|
|
.map_err(HotColdDBError::RestorePointBlockHashError)
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/// Load the blocks between `start_slot` and `end_slot` by backtracking from `end_block_hash`.
|
|
|
|
///
|
|
|
|
/// Blocks are returned in slot-ascending order, suitable for replaying on a state with slot
|
|
|
|
/// equal to `start_slot`, to reach a state with slot equal to `end_slot`.
|
2019-12-06 07:52:11 +00:00
|
|
|
fn load_blocks_to_replay(
|
2019-12-06 03:29:06 +00:00
|
|
|
&self,
|
|
|
|
start_slot: Slot,
|
|
|
|
end_slot: Slot,
|
|
|
|
end_block_hash: Hash256,
|
2020-02-10 23:19:36 +00:00
|
|
|
) -> Result<Vec<SignedBeaconBlock<E>>, Error> {
|
2020-05-16 03:23:32 +00:00
|
|
|
let mut blocks: Vec<SignedBeaconBlock<E>> =
|
|
|
|
ParentRootBlockIterator::new(self, end_block_hash)
|
|
|
|
.map(|result| result.map(|(_, block)| block))
|
|
|
|
// Include the block at the end slot (if any), it needs to be
|
|
|
|
// replayed in order to construct the canonical state at `end_slot`.
|
|
|
|
.filter(|result| {
|
|
|
|
result
|
|
|
|
.as_ref()
|
2021-07-09 06:15:32 +00:00
|
|
|
.map_or(true, |block| block.slot() <= end_slot)
|
2020-05-16 03:23:32 +00:00
|
|
|
})
|
|
|
|
// Include the block at the start slot (if any). Whilst it doesn't need to be applied
|
|
|
|
// to the state, it contains a potentially useful state root.
|
|
|
|
.take_while(|result| {
|
|
|
|
result
|
|
|
|
.as_ref()
|
2021-07-09 06:15:32 +00:00
|
|
|
.map_or(true, |block| block.slot() >= start_slot)
|
2020-05-16 03:23:32 +00:00
|
|
|
})
|
|
|
|
.collect::<Result<_, _>>()?;
|
2019-12-06 03:29:06 +00:00
|
|
|
blocks.reverse();
|
|
|
|
Ok(blocks)
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Replay `blocks` on top of `state` until `target_slot` is reached.
|
|
|
|
///
|
2020-01-09 00:43:11 +00:00
|
|
|
/// Will skip slots as necessary. The returned state is not guaranteed
|
|
|
|
/// to have any caches built, beyond those immediately required by block processing.
|
2019-12-06 07:52:11 +00:00
|
|
|
fn replay_blocks(
|
2019-12-06 03:29:06 +00:00
|
|
|
&self,
|
|
|
|
mut state: BeaconState<E>,
|
2020-08-17 08:05:13 +00:00
|
|
|
mut blocks: Vec<SignedBeaconBlock<E>>,
|
2019-12-06 03:29:06 +00:00
|
|
|
target_slot: Slot,
|
2020-08-17 08:05:13 +00:00
|
|
|
block_replay: BlockReplay,
|
2019-12-06 03:29:06 +00:00
|
|
|
) -> Result<BeaconState<E>, Error> {
|
2020-08-17 08:05:13 +00:00
|
|
|
if block_replay == BlockReplay::InconsistentStateRoots {
|
|
|
|
for i in 0..blocks.len() {
|
2021-07-09 06:15:32 +00:00
|
|
|
let prev_block_root = if i > 0 {
|
|
|
|
blocks[i - 1].canonical_root()
|
|
|
|
} else {
|
|
|
|
// Not read.
|
|
|
|
Hash256::zero()
|
|
|
|
};
|
|
|
|
|
|
|
|
let (state_root, parent_root) = match &mut blocks[i] {
|
|
|
|
SignedBeaconBlock::Base(block) => (
|
|
|
|
&mut block.message.state_root,
|
|
|
|
&mut block.message.parent_root,
|
|
|
|
),
|
|
|
|
SignedBeaconBlock::Altair(block) => (
|
|
|
|
&mut block.message.state_root,
|
|
|
|
&mut block.message.parent_root,
|
|
|
|
),
|
|
|
|
};
|
|
|
|
|
|
|
|
*state_root = Hash256::zero();
|
2020-08-17 08:05:13 +00:00
|
|
|
if i > 0 {
|
2021-07-09 06:15:32 +00:00
|
|
|
*parent_root = prev_block_root;
|
2020-08-17 08:05:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
let state_root_from_prev_block = |i: usize, state: &BeaconState<E>| {
|
|
|
|
if i > 0 {
|
2021-07-09 06:15:32 +00:00
|
|
|
let prev_block = blocks[i - 1].message();
|
|
|
|
if prev_block.slot() == state.slot() {
|
|
|
|
Some(prev_block.state_root())
|
2020-01-08 02:58:01 +00:00
|
|
|
} else {
|
|
|
|
None
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
None
|
|
|
|
}
|
|
|
|
};
|
2019-12-06 03:29:06 +00:00
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
for (i, block) in blocks.iter().enumerate() {
|
2021-07-09 06:15:32 +00:00
|
|
|
if block.slot() <= state.slot() {
|
2020-03-02 02:40:58 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2021-07-09 06:15:32 +00:00
|
|
|
while state.slot() < block.slot() {
|
2020-08-17 08:05:13 +00:00
|
|
|
let state_root = match block_replay {
|
|
|
|
BlockReplay::Accurate => state_root_from_prev_block(i, &state),
|
|
|
|
BlockReplay::InconsistentStateRoots => Some(Hash256::zero()),
|
|
|
|
};
|
2020-01-08 02:58:01 +00:00
|
|
|
per_slot_processing(&mut state, state_root, &self.spec)
|
|
|
|
.map_err(HotColdDBError::BlockReplaySlotError)?;
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
2020-08-17 08:05:13 +00:00
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
per_block_processing(
|
|
|
|
&mut state,
|
|
|
|
&block,
|
|
|
|
None,
|
|
|
|
BlockSignatureStrategy::NoVerification,
|
|
|
|
&self.spec,
|
|
|
|
)
|
2020-01-08 02:58:01 +00:00
|
|
|
.map_err(HotColdDBError::BlockReplayBlockError)?;
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
2019-11-26 23:54:46 +00:00
|
|
|
|
2021-07-09 06:15:32 +00:00
|
|
|
while state.slot() < target_slot {
|
2020-08-17 08:05:13 +00:00
|
|
|
let state_root = match block_replay {
|
|
|
|
BlockReplay::Accurate => state_root_from_prev_block(blocks.len(), &state),
|
|
|
|
BlockReplay::InconsistentStateRoots => Some(Hash256::zero()),
|
|
|
|
};
|
2020-01-08 02:58:01 +00:00
|
|
|
per_slot_processing(&mut state, state_root, &self.spec)
|
|
|
|
.map_err(HotColdDBError::BlockReplaySlotError)?;
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
Ok(state)
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Fetch a copy of the current split slot from memory.
|
2019-11-26 23:54:46 +00:00
|
|
|
pub fn get_split_slot(&self) -> Slot {
|
2019-12-06 03:29:06 +00:00
|
|
|
self.split.read().slot
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
2019-12-06 07:52:11 +00:00
|
|
|
/// Fetch the slot of the most recently stored restore point.
|
|
|
|
pub fn get_latest_restore_point_slot(&self) -> Slot {
|
2020-02-10 00:30:21 +00:00
|
|
|
(self.get_split_slot() - 1) / self.config.slots_per_restore_point
|
|
|
|
* self.config.slots_per_restore_point
|
2019-12-06 07:52:11 +00:00
|
|
|
}
|
|
|
|
|
2020-09-30 02:36:07 +00:00
|
|
|
/// Load the database schema version from disk.
|
|
|
|
fn load_schema_version(&self) -> Result<Option<SchemaVersion>, Error> {
|
|
|
|
self.hot_db.get(&SCHEMA_VERSION_KEY)
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Store the database schema version.
|
2021-03-04 01:25:12 +00:00
|
|
|
pub fn store_schema_version(&self, schema_version: SchemaVersion) -> Result<(), Error> {
|
2020-09-30 02:36:07 +00:00
|
|
|
self.hot_db.put(&SCHEMA_VERSION_KEY, &schema_version)
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Load previously-stored config from disk.
|
2020-11-17 09:10:53 +00:00
|
|
|
fn load_config(&self) -> Result<Option<OnDiskStoreConfig>, Error> {
|
2020-09-30 02:36:07 +00:00
|
|
|
self.hot_db.get(&CONFIG_KEY)
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Write the config to disk.
|
|
|
|
fn store_config(&self) -> Result<(), Error> {
|
2020-11-17 09:10:53 +00:00
|
|
|
self.hot_db.put(&CONFIG_KEY, &self.config.as_disk_config())
|
2020-09-30 02:36:07 +00:00
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Load the split point from disk.
|
|
|
|
fn load_split(&self) -> Result<Option<Split>, Error> {
|
2020-09-30 02:36:07 +00:00
|
|
|
self.hot_db.get(&SPLIT_KEY)
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Load the state root of a restore point.
|
|
|
|
fn load_restore_point_hash(&self, restore_point_index: u64) -> Result<Hash256, Error> {
|
|
|
|
let key = Self::restore_point_key(restore_point_index);
|
2020-05-25 00:26:54 +00:00
|
|
|
self.cold_db
|
|
|
|
.get(&key)?
|
|
|
|
.map(|r: RestorePointHash| r.state_root)
|
2020-01-21 07:38:56 +00:00
|
|
|
.ok_or_else(|| HotColdDBError::MissingRestorePointHash(restore_point_index).into())
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/// Store the state root of a restore point.
|
|
|
|
fn store_restore_point_hash(
|
|
|
|
&self,
|
|
|
|
restore_point_index: u64,
|
|
|
|
state_root: Hash256,
|
2020-07-01 02:45:57 +00:00
|
|
|
ops: &mut Vec<KeyValueStoreOp>,
|
|
|
|
) {
|
|
|
|
let value = &RestorePointHash { state_root };
|
|
|
|
let op = value.as_kv_store_op(Self::restore_point_key(restore_point_index));
|
|
|
|
ops.push(op);
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/// Convert a `restore_point_index` into a database key.
|
|
|
|
fn restore_point_key(restore_point_index: u64) -> Hash256 {
|
|
|
|
Hash256::from_low_u64_be(restore_point_index)
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Load a frozen state's slot, given its root.
|
2020-01-08 02:58:01 +00:00
|
|
|
fn load_cold_state_slot(&self, state_root: &Hash256) -> Result<Option<Slot>, Error> {
|
2020-05-25 00:26:54 +00:00
|
|
|
Ok(self
|
|
|
|
.cold_db
|
|
|
|
.get(state_root)?
|
|
|
|
.map(|s: ColdStateSummary| s.slot))
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Load a hot state's summary, given its root.
|
|
|
|
pub fn load_hot_state_summary(
|
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
) -> Result<Option<HotStateSummary>, Error> {
|
2020-05-25 00:26:54 +00:00
|
|
|
self.hot_db.get(state_root)
|
2020-01-08 02:58:01 +00:00
|
|
|
}
|
|
|
|
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
/// Load the temporary flag for a state root, if one exists.
|
|
|
|
///
|
|
|
|
/// Returns `Some` if the state is temporary, or `None` if the state is permanent or does not
|
|
|
|
/// exist -- you should call `load_hot_state_summary` to find out which.
|
|
|
|
pub fn load_state_temporary_flag(
|
|
|
|
&self,
|
|
|
|
state_root: &Hash256,
|
|
|
|
) -> Result<Option<TemporaryFlag>, Error> {
|
|
|
|
self.hot_db.get(state_root)
|
|
|
|
}
|
|
|
|
|
2020-01-09 10:05:56 +00:00
|
|
|
/// Check that the restore point frequency is valid.
|
2019-12-06 03:29:06 +00:00
|
|
|
///
|
2020-01-09 10:05:56 +00:00
|
|
|
/// Specifically, check that it is:
|
|
|
|
/// (1) A divisor of the number of slots per historical root, and
|
|
|
|
/// (2) Divisible by the number of slots per epoch
|
|
|
|
///
|
|
|
|
///
|
|
|
|
/// (1) ensures that we have at least one restore point within range of our state
|
2019-12-06 03:29:06 +00:00
|
|
|
/// root history when iterating backwards (and allows for more frequent restore points if
|
|
|
|
/// desired).
|
2020-01-09 10:05:56 +00:00
|
|
|
///
|
|
|
|
/// (2) ensures that restore points align with hot state summaries, making it
|
|
|
|
/// quick to migrate hot to cold.
|
2020-01-08 02:58:01 +00:00
|
|
|
fn verify_slots_per_restore_point(slots_per_restore_point: u64) -> Result<(), HotColdDBError> {
|
2019-12-06 03:29:06 +00:00
|
|
|
let slots_per_historical_root = E::SlotsPerHistoricalRoot::to_u64();
|
2020-01-09 10:05:56 +00:00
|
|
|
let slots_per_epoch = E::slots_per_epoch();
|
|
|
|
if slots_per_restore_point > 0
|
|
|
|
&& slots_per_historical_root % slots_per_restore_point == 0
|
|
|
|
&& slots_per_restore_point % slots_per_epoch == 0
|
|
|
|
{
|
2019-12-06 03:29:06 +00:00
|
|
|
Ok(())
|
|
|
|
} else {
|
2020-01-08 02:58:01 +00:00
|
|
|
Err(HotColdDBError::InvalidSlotsPerRestorePoint {
|
2019-12-06 03:29:06 +00:00
|
|
|
slots_per_restore_point,
|
|
|
|
slots_per_historical_root,
|
2020-01-09 10:05:56 +00:00
|
|
|
slots_per_epoch,
|
2019-12-06 03:29:06 +00:00
|
|
|
})
|
|
|
|
}
|
|
|
|
}
|
2020-11-09 07:02:21 +00:00
|
|
|
|
|
|
|
/// Run a compaction pass to free up space used by deleted states.
|
|
|
|
pub fn compact(&self) -> Result<(), Error> {
|
|
|
|
self.hot_db.compact()?;
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
2020-11-17 09:10:53 +00:00
|
|
|
/// Return `true` if compaction on finalization/pruning is enabled.
|
|
|
|
pub fn compact_on_prune(&self) -> bool {
|
|
|
|
self.config.compact_on_prune
|
|
|
|
}
|
|
|
|
|
2020-11-09 07:02:21 +00:00
|
|
|
/// Load the checkpoint to begin pruning from (the "old finalized checkpoint").
|
|
|
|
pub fn load_pruning_checkpoint(&self) -> Result<Option<Checkpoint>, Error> {
|
|
|
|
Ok(self
|
|
|
|
.hot_db
|
|
|
|
.get(&PRUNING_CHECKPOINT_KEY)?
|
|
|
|
.map(|pc: PruningCheckpoint| pc.checkpoint))
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Create a staged store for the pruning checkpoint.
|
|
|
|
pub fn pruning_checkpoint_store_op(&self, checkpoint: Checkpoint) -> KeyValueStoreOp {
|
|
|
|
PruningCheckpoint { checkpoint }.as_kv_store_op(PRUNING_CHECKPOINT_KEY)
|
|
|
|
}
|
2020-11-17 09:10:53 +00:00
|
|
|
|
|
|
|
/// Load the timestamp of the last compaction as a `Duration` since the UNIX epoch.
|
|
|
|
pub fn load_compaction_timestamp(&self) -> Result<Option<Duration>, Error> {
|
|
|
|
Ok(self
|
|
|
|
.hot_db
|
|
|
|
.get(&COMPACTION_TIMESTAMP_KEY)?
|
|
|
|
.map(|c: CompactionTimestamp| Duration::from_secs(c.0)))
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Store the timestamp of the last compaction as a `Duration` since the UNIX epoch.
|
|
|
|
pub fn store_compaction_timestamp(&self, compaction_timestamp: Duration) -> Result<(), Error> {
|
|
|
|
self.hot_db.put(
|
|
|
|
&COMPACTION_TIMESTAMP_KEY,
|
|
|
|
&CompactionTimestamp(compaction_timestamp.as_secs()),
|
|
|
|
)
|
|
|
|
}
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
/// Advance the split point of the store, moving new finalized states to the freezer.
|
2020-08-26 09:24:55 +00:00
|
|
|
pub fn migrate_database<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>>(
|
2020-06-16 01:34:04 +00:00
|
|
|
store: Arc<HotColdDB<E, Hot, Cold>>,
|
2020-05-31 22:13:49 +00:00
|
|
|
frozen_head_root: Hash256,
|
|
|
|
frozen_head: &BeaconState<E>,
|
|
|
|
) -> Result<(), Error> {
|
|
|
|
debug!(
|
|
|
|
store.log,
|
|
|
|
"Freezer migration started";
|
2021-07-09 06:15:32 +00:00
|
|
|
"slot" => frozen_head.slot()
|
2020-05-31 22:13:49 +00:00
|
|
|
);
|
|
|
|
|
|
|
|
// 0. Check that the migration is sensible.
|
|
|
|
// The new frozen head must increase the current split slot, and lie on an epoch
|
|
|
|
// boundary (in order for the hot state summary scheme to work).
|
2020-07-02 23:47:31 +00:00
|
|
|
let current_split_slot = store.split.read().slot;
|
2020-05-31 22:13:49 +00:00
|
|
|
|
2021-07-09 06:15:32 +00:00
|
|
|
if frozen_head.slot() < current_split_slot {
|
2020-05-31 22:13:49 +00:00
|
|
|
return Err(HotColdDBError::FreezeSlotError {
|
|
|
|
current_split_slot,
|
2021-07-09 06:15:32 +00:00
|
|
|
proposed_split_slot: frozen_head.slot(),
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
|
|
|
.into());
|
|
|
|
}
|
|
|
|
|
2021-07-09 06:15:32 +00:00
|
|
|
if frozen_head.slot() % E::slots_per_epoch() != 0 {
|
|
|
|
return Err(HotColdDBError::FreezeSlotUnaligned(frozen_head.slot()).into());
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
|
|
|
|
2020-07-02 23:47:31 +00:00
|
|
|
let mut hot_db_ops: Vec<StoreOp<E>> = Vec::new();
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
// 1. Copy all of the states between the head and the split slot, from the hot DB
|
|
|
|
// to the cold DB.
|
|
|
|
let state_root_iter = StateRootsIterator::new(store.clone(), frozen_head);
|
2020-06-09 23:55:44 +00:00
|
|
|
for maybe_pair in state_root_iter.take_while(|result| match result {
|
|
|
|
Ok((_, slot)) => slot >= ¤t_split_slot,
|
|
|
|
Err(_) => true,
|
|
|
|
}) {
|
|
|
|
let (state_root, slot) = maybe_pair?;
|
2020-07-02 23:47:31 +00:00
|
|
|
|
|
|
|
let mut cold_db_ops: Vec<KeyValueStoreOp> = Vec::new();
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
if slot % store.config.slots_per_restore_point == 0 {
|
2021-07-09 06:15:32 +00:00
|
|
|
let state: BeaconState<E> = get_full_state(&store.hot_db, &state_root, &store.spec)?
|
2020-12-03 01:10:26 +00:00
|
|
|
.ok_or(HotColdDBError::MissingStateToFreeze(state_root))?;
|
2020-05-31 22:13:49 +00:00
|
|
|
|
2020-07-02 23:47:31 +00:00
|
|
|
store.store_cold_state(&state_root, &state, &mut cold_db_ops)?;
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Store a pointer from this state root to its slot, so we can later reconstruct states
|
|
|
|
// from their state root alone.
|
2020-07-02 23:47:31 +00:00
|
|
|
let cold_state_summary = ColdStateSummary { slot };
|
|
|
|
let op = cold_state_summary.as_kv_store_op(state_root);
|
|
|
|
cold_db_ops.push(op);
|
|
|
|
|
|
|
|
// There are data dependencies between calls to `store_cold_state()` that prevent us from
|
|
|
|
// doing one big call to `store.cold_db.do_atomically()` at end of the loop.
|
|
|
|
store.cold_db.do_atomically(cold_db_ops)?;
|
2020-05-31 22:13:49 +00:00
|
|
|
|
|
|
|
// Delete the old summary, and the full state if we lie on an epoch boundary.
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
hot_db_ops.push(StoreOp::DeleteState(state_root, Some(slot)));
|
2020-07-02 23:47:31 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Warning: Critical section. We have to take care not to put any of the two databases in an
|
|
|
|
// inconsistent state if the OS process dies at any point during the freezeing
|
|
|
|
// procedure.
|
|
|
|
//
|
|
|
|
// Since it is pretty much impossible to be atomic across more than one database, we trade
|
|
|
|
// losing track of states to delete, for consistency. In other words: We should be safe to die
|
|
|
|
// at any point below but it may happen that some states won't be deleted from the hot database
|
|
|
|
// and will remain there forever. Since dying in these particular few lines should be an
|
|
|
|
// exceedingly rare event, this should be an acceptable tradeoff.
|
|
|
|
|
|
|
|
// Flush to disk all the states that have just been migrated to the cold store.
|
|
|
|
store.cold_db.sync()?;
|
|
|
|
|
|
|
|
{
|
|
|
|
let mut split_guard = store.split.write();
|
|
|
|
let latest_split_slot = split_guard.slot;
|
|
|
|
|
|
|
|
// Detect a sitation where the split point is (erroneously) changed from more than one
|
|
|
|
// place in code.
|
|
|
|
if latest_split_slot != current_split_slot {
|
|
|
|
error!(
|
|
|
|
store.log,
|
|
|
|
"Race condition detected: Split point changed while moving states to the freezer";
|
|
|
|
"previous split slot" => current_split_slot,
|
|
|
|
"current split slot" => latest_split_slot,
|
|
|
|
);
|
2020-05-31 22:13:49 +00:00
|
|
|
|
2020-07-02 23:47:31 +00:00
|
|
|
// Assume the freezing procedure will be retried in case this happens.
|
|
|
|
return Err(Error::SplitPointModified(
|
|
|
|
current_split_slot,
|
|
|
|
latest_split_slot,
|
|
|
|
));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Before updating the in-memory split value, we flush it to disk first, so that should the
|
|
|
|
// OS process die at this point, we pick up from the right place after a restart.
|
|
|
|
let split = Split {
|
2021-07-09 06:15:32 +00:00
|
|
|
slot: frozen_head.slot(),
|
2020-07-02 23:47:31 +00:00
|
|
|
state_root: frozen_head_root,
|
|
|
|
};
|
2020-09-30 02:36:07 +00:00
|
|
|
store.hot_db.put_sync(&SPLIT_KEY, &split)?;
|
2020-05-31 22:13:49 +00:00
|
|
|
|
2020-07-02 23:47:31 +00:00
|
|
|
// Split point is now persisted in the hot database on disk. The in-memory split point
|
|
|
|
// hasn't been modified elsewhere since we keep a write lock on it. It's safe to update
|
|
|
|
// the in-memory split point now.
|
|
|
|
*split_guard = split;
|
2020-05-31 22:13:49 +00:00
|
|
|
}
|
|
|
|
|
2020-07-02 23:47:31 +00:00
|
|
|
// Delete the states from the hot database if we got this far.
|
|
|
|
store.do_atomically(hot_db_ops)?;
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
debug!(
|
|
|
|
store.log,
|
|
|
|
"Freezer migration complete";
|
2021-07-09 06:15:32 +00:00
|
|
|
"slot" => frozen_head.slot()
|
2020-05-31 22:13:49 +00:00
|
|
|
);
|
|
|
|
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
2019-12-06 03:29:06 +00:00
|
|
|
/// Struct for storing the split slot and state root in the database.
|
2020-01-08 02:58:01 +00:00
|
|
|
#[derive(Debug, Clone, Copy, Default, Encode, Decode)]
|
2020-07-02 23:47:31 +00:00
|
|
|
pub struct Split {
|
2019-12-06 03:29:06 +00:00
|
|
|
slot: Slot,
|
|
|
|
state_root: Hash256,
|
|
|
|
}
|
2019-11-26 23:54:46 +00:00
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
impl StoreItem for Split {
|
2019-11-26 23:54:46 +00:00
|
|
|
fn db_column() -> DBColumn {
|
|
|
|
DBColumn::BeaconMeta
|
|
|
|
}
|
|
|
|
|
|
|
|
fn as_store_bytes(&self) -> Vec<u8> {
|
2019-12-06 03:29:06 +00:00
|
|
|
self.as_ssz_bytes()
|
|
|
|
}
|
|
|
|
|
|
|
|
fn from_store_bytes(bytes: &[u8]) -> Result<Self, Error> {
|
|
|
|
Ok(Self::from_ssz_bytes(bytes)?)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Struct for summarising a state in the hot database.
|
|
|
|
///
|
|
|
|
/// Allows full reconstruction by replaying blocks.
|
|
|
|
#[derive(Debug, Clone, Copy, Default, Encode, Decode)]
|
|
|
|
pub struct HotStateSummary {
|
|
|
|
slot: Slot,
|
|
|
|
latest_block_root: Hash256,
|
|
|
|
epoch_boundary_state_root: Hash256,
|
|
|
|
}
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
impl StoreItem for HotStateSummary {
|
2020-01-08 02:58:01 +00:00
|
|
|
fn db_column() -> DBColumn {
|
|
|
|
DBColumn::BeaconStateSummary
|
|
|
|
}
|
|
|
|
|
|
|
|
fn as_store_bytes(&self) -> Vec<u8> {
|
|
|
|
self.as_ssz_bytes()
|
|
|
|
}
|
|
|
|
|
|
|
|
fn from_store_bytes(bytes: &[u8]) -> Result<Self, Error> {
|
|
|
|
Ok(Self::from_ssz_bytes(bytes)?)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-02-10 00:30:21 +00:00
|
|
|
impl HotStateSummary {
|
|
|
|
/// Construct a new summary of the given state.
|
|
|
|
pub fn new<E: EthSpec>(state_root: &Hash256, state: &BeaconState<E>) -> Result<Self, Error> {
|
|
|
|
// Fill in the state root on the latest block header if necessary (this happens on all
|
|
|
|
// slots where there isn't a skip).
|
|
|
|
let latest_block_root = state.get_latest_block_root(*state_root);
|
2021-07-09 06:15:32 +00:00
|
|
|
let epoch_boundary_slot = state.slot() / E::slots_per_epoch() * E::slots_per_epoch();
|
|
|
|
let epoch_boundary_state_root = if epoch_boundary_slot == state.slot() {
|
2020-02-10 00:30:21 +00:00
|
|
|
*state_root
|
|
|
|
} else {
|
|
|
|
*state
|
|
|
|
.get_state_root(epoch_boundary_slot)
|
|
|
|
.map_err(HotColdDBError::HotStateSummaryError)?
|
|
|
|
};
|
|
|
|
|
|
|
|
Ok(HotStateSummary {
|
2021-07-09 06:15:32 +00:00
|
|
|
slot: state.slot(),
|
2020-02-10 00:30:21 +00:00
|
|
|
latest_block_root,
|
|
|
|
epoch_boundary_state_root,
|
|
|
|
})
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-01-08 02:58:01 +00:00
|
|
|
/// Struct for summarising a state in the freezer database.
|
|
|
|
#[derive(Debug, Clone, Copy, Default, Encode, Decode)]
|
|
|
|
struct ColdStateSummary {
|
2019-12-06 03:29:06 +00:00
|
|
|
slot: Slot,
|
|
|
|
}
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
impl StoreItem for ColdStateSummary {
|
2019-12-06 03:29:06 +00:00
|
|
|
fn db_column() -> DBColumn {
|
2020-01-08 02:58:01 +00:00
|
|
|
DBColumn::BeaconStateSummary
|
2019-12-06 03:29:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
fn as_store_bytes(&self) -> Vec<u8> {
|
|
|
|
self.as_ssz_bytes()
|
|
|
|
}
|
|
|
|
|
|
|
|
fn from_store_bytes(bytes: &[u8]) -> Result<Self, Error> {
|
|
|
|
Ok(Self::from_ssz_bytes(bytes)?)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Struct for storing the state root of a restore point in the database.
|
2020-01-08 02:58:01 +00:00
|
|
|
#[derive(Debug, Clone, Copy, Default, Encode, Decode)]
|
2019-12-06 03:29:06 +00:00
|
|
|
struct RestorePointHash {
|
|
|
|
state_root: Hash256,
|
|
|
|
}
|
|
|
|
|
2020-05-31 22:13:49 +00:00
|
|
|
impl StoreItem for RestorePointHash {
|
2019-12-06 03:29:06 +00:00
|
|
|
fn db_column() -> DBColumn {
|
|
|
|
DBColumn::BeaconRestorePoint
|
|
|
|
}
|
|
|
|
|
|
|
|
fn as_store_bytes(&self) -> Vec<u8> {
|
|
|
|
self.as_ssz_bytes()
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
fn from_store_bytes(bytes: &[u8]) -> Result<Self, Error> {
|
2019-12-06 03:29:06 +00:00
|
|
|
Ok(Self::from_ssz_bytes(bytes)?)
|
2019-11-26 23:54:46 +00:00
|
|
|
}
|
|
|
|
}
|
Implement database temp states to reduce memory usage (#1798)
## Issue Addressed
Closes #800
Closes #1713
## Proposed Changes
Implement the temporary state storage algorithm described in #800. Specifically:
* Add `DBColumn::BeaconStateTemporary`, for storing 0-length temporary marker values.
* Store intermediate states immediately as they are created, marked temporary. Delete the temporary flag if the block is processed successfully.
* Add a garbage collection process to delete leftover temporary states on start-up.
* Bump the database schema version to 2 so that a DB with temporary states can't accidentally be used with older versions of the software. The auto-migration is a no-op, but puts in place some infra that we can use for future migrations (e.g. #1784)
## Additional Info
There are two known race conditions, one potentially causing permanent faults (hopefully rare), and the other insignificant.
### Race 1: Permanent state marked temporary
EDIT: this has been fixed by the addition of a lock around the relevant critical section
There are 2 threads that are trying to store 2 different blocks that share some intermediate states (e.g. they both skip some slots from the current head). Consider this sequence of events:
1. Thread 1 checks if state `s` already exists, and seeing that it doesn't, prepares an atomic commit of `(s, s_temporary_flag)`.
2. Thread 2 does the same, but also gets as far as committing the state txn, finishing the processing of its block, and _deleting_ the temporary flag.
3. Thread 1 is (finally) scheduled again, and marks `s` as temporary with its transaction.
4.
a) The process is killed, or thread 1's block fails verification and the temp flag is not deleted. This is a permanent failure! Any attempt to load state `s` will fail... hope it isn't on the main chain! Alternatively (4b) happens...
b) Thread 1 finishes, and re-deletes the temporary flag. In this case the failure is transient, state `s` will disappear temporarily, but will come back once thread 1 finishes running.
I _hope_ that steps 1-3 only happen very rarely, and 4a even more rarely. It's hard to know
This once again begs the question of why we're using LevelDB (#483), when it clearly doesn't care about atomicity! A ham-fisted fix would be to wrap the hot and cold DBs in locks, which would bring us closer to how other DBs handle read-write transactions. E.g. [LMDB only allows one R/W transaction at a time](https://docs.rs/lmdb/0.8.0/lmdb/struct.Environment.html#method.begin_rw_txn).
### Race 2: Temporary state returned from `get_state`
I don't think this race really matters, but in `load_hot_state`, if another thread stores a state between when we call `load_state_temporary_flag` and when we call `load_hot_state_summary`, then we could end up returning that state even though it's only a temporary state. I can't think of any case where this would be relevant, and I suspect if it did come up, it would be safe/recoverable (having data is safer than _not_ having data).
This could be fixed by using a LevelDB read snapshot, but that would require substantial changes to how we read all our values, so I don't think it's worth it right now.
2020-10-23 01:27:51 +00:00
|
|
|
|
|
|
|
#[derive(Debug, Clone, Copy, Default)]
|
|
|
|
pub struct TemporaryFlag;
|
|
|
|
|
|
|
|
impl StoreItem for TemporaryFlag {
|
|
|
|
fn db_column() -> DBColumn {
|
|
|
|
DBColumn::BeaconStateTemporary
|
|
|
|
}
|
|
|
|
|
|
|
|
fn as_store_bytes(&self) -> Vec<u8> {
|
|
|
|
vec![]
|
|
|
|
}
|
|
|
|
|
|
|
|
fn from_store_bytes(_: &[u8]) -> Result<Self, Error> {
|
|
|
|
Ok(TemporaryFlag)
|
|
|
|
}
|
|
|
|
}
|