7d644103c6
## Issue Addressed Resolves #1890 ## Proposed Changes Change the slasher database schema to key indexed attestations by `(target_epoch, indexed_attestation_root)` instead of just `indexed_attestation_root`. This allows more straight-forward pruning (linear scan), that is also "re-entrant". By re-entrant, we mean that a pruning pass that gets stuck because of a `MapFull` error can attempt to commit midway, and be resumed later without issue. The previous pruning strategy for indexed attestations did not have this property. There was also a flaw in the previous pruning that could leave "zombie" indexed attestations in the database (ones not referenced by any attester record), which could build up and contribute to bloat (although in practice I think they occur quite infrequently). ## Additional Info During testing I noticed that a `MapFull` error can still occur during the commit of the transaction itself, which is irritating, but not unbearable. This PR should at least reduce the frequency with which users need to manually resize their DB, and if the `MapFull` on commit rears its ugly head too often we could use a dynamic strategy (temporarily increase the size of the map until the transaction commits). The extra bytes for the epoch make the database a bit heavier, so the size estimate docs have been updated to reflect this. This is also a breaking schema change, so anyone using a v0 database from a few hours ago will need to drop it and update 😅
90 lines
2.5 KiB
Rust
90 lines
2.5 KiB
Rust
use slasher::{
|
|
test_utils::{indexed_att, logger},
|
|
Config, Error, Slasher,
|
|
};
|
|
use tempdir::TempDir;
|
|
use types::Epoch;
|
|
|
|
#[test]
|
|
fn attestation_pruning_empty_wrap_around() {
|
|
let tempdir = TempDir::new("slasher").unwrap();
|
|
let mut config = Config::new(tempdir.path().into());
|
|
config.validator_chunk_size = 1;
|
|
config.chunk_size = 16;
|
|
config.history_length = 16;
|
|
|
|
let slasher = Slasher::open(config.clone(), logger()).unwrap();
|
|
|
|
let v = vec![0];
|
|
let history_length = config.history_length as u64;
|
|
|
|
let mut current_epoch = Epoch::new(history_length - 1);
|
|
|
|
slasher.accept_attestation(indexed_att(v.clone(), 0, history_length - 1, 0));
|
|
slasher.process_queued(current_epoch).unwrap();
|
|
slasher.prune_database(current_epoch).unwrap();
|
|
|
|
// Delete the previous attestation
|
|
current_epoch = Epoch::new(2 * history_length + 2);
|
|
slasher.prune_database(current_epoch).unwrap();
|
|
|
|
// Add an attestation that would be surrounded with the modulo considered
|
|
slasher.accept_attestation(indexed_att(
|
|
v.clone(),
|
|
2 * history_length - 3,
|
|
2 * history_length - 2,
|
|
1,
|
|
));
|
|
slasher.process_queued(current_epoch).unwrap();
|
|
}
|
|
|
|
// Test that pruning can recover from a `MapFull` error
|
|
#[test]
|
|
fn pruning_with_map_full() {
|
|
let tempdir = TempDir::new("slasher").unwrap();
|
|
let mut config = Config::new(tempdir.path().into());
|
|
config.validator_chunk_size = 1;
|
|
config.chunk_size = 16;
|
|
config.history_length = 1024;
|
|
config.max_db_size_mbs = 1;
|
|
|
|
let slasher = Slasher::open(config.clone(), logger()).unwrap();
|
|
|
|
let v = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12];
|
|
|
|
let mut current_epoch = Epoch::new(0);
|
|
|
|
loop {
|
|
slasher.accept_attestation(indexed_att(
|
|
v.clone(),
|
|
(current_epoch - 1).as_u64(),
|
|
current_epoch.as_u64(),
|
|
0,
|
|
));
|
|
if let Err(Error::DatabaseError(lmdb::Error::MapFull)) =
|
|
slasher.process_queued(current_epoch)
|
|
{
|
|
break;
|
|
}
|
|
current_epoch += 1;
|
|
}
|
|
|
|
loop {
|
|
slasher.prune_database(current_epoch).unwrap();
|
|
|
|
slasher.accept_attestation(indexed_att(
|
|
v.clone(),
|
|
(current_epoch - 1).as_u64(),
|
|
current_epoch.as_u64(),
|
|
0,
|
|
));
|
|
match slasher.process_queued(current_epoch) {
|
|
Ok(()) => break,
|
|
Err(Error::DatabaseError(lmdb::Error::MapFull)) => {
|
|
current_epoch += 1;
|
|
}
|
|
Err(e) => panic!("{:?}", e),
|
|
}
|
|
}
|
|
}
|