Commit Graph

243 Commits

Author SHA1 Message Date
Diva M
d93753cc88
Merge branch 'unstable' into off-4844 2023-03-02 15:38:00 -05:00
Michael Sproul
0b6850221e
Fix Capella schema downgrades (#4004) 2023-02-20 17:50:42 +11:00
Michael Sproul
18c8cab4da
Merge remote-tracking branch 'origin/unstable' into capella-merge 2023-02-14 12:07:27 +11:00
realbigsean
28702c9d5d
merge upstream, add back get_blobs logic 2023-02-13 16:29:21 -05:00
Michael Sproul
2f456ff9eb Fix regression in DB write atomicity (#3931)
## Issue Addressed

Fix a bug introduced by #3696. The bug is not expected to occur frequently, so releasing this PR is non-urgent.

## Proposed Changes

* Add a variant to `StoreOp` that allows a raw KV operation to be passed around.
* Return to using `self.store.do_atomically` rather than `self.store.hot_db.do_atomically`. This streamlines the write back into a single call and makes our auto-revert work again.
* Prevent `import_block_update_shuffling_cache` from failing block import. This is an outstanding bug from before v3.4.0 which may have contributed to some random unexplained database corruption.

## Additional Info

In #3696 I split the database write into two calls, one to convert the `StoreOp`s to `KeyValueStoreOp`s and one to write them. This had the unfortunate side-effect of damaging our atomicity guarantees in case of a write error. If the first call failed, we would be left with the block in fork choice but not on-disk (or the snapshot cache), which would prevent us from processing any descendant blocks. On `unstable` the first call is very unlikely to fail unless the disk is full, but on `tree-states` the conversion is more involved and a user reported database corruption after it failed in a way that should have been recoverable.

Additionally, as @emhane observed, #3696 also inadvertently removed the import of the new block into the block cache. Although this seems like it could have negatively impacted performance, there are several mitigating factors:

- For regular block processing we should almost always load the parent block (and state) from the snapshot cache.
- We often load blinded blocks, which bypass the block cache anyway.
- Metrics show no noticeable increase in the block cache miss rate with v3.4.0.

However, I expect the block cache _will_ be useful again in `tree-states`, so it is restored to use by this PR.
2023-02-13 03:32:01 +00:00
Emilia Hane
28e9f07746
Fix lint for prune blobs pr 2023-02-10 16:23:04 +01:00
Emilia Hane
6a37e84399
fixup! Fix regression in DB write atomicity 2023-02-08 11:44:46 +01:00
Michael Sproul
ac4b5b580c
Fix regression in DB write atomicity 2023-02-08 11:44:45 +01:00
Emilia Hane
9d919917f5
Removed unused code 2023-02-08 11:44:45 +01:00
Emilia Hane
d7eb9441cf
Reorder loading of db metadata from disk to allow for future changes to schema 2023-02-08 11:44:45 +01:00
Emilia Hane
d599e41f3d
Remove debug comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
577262ccbf
Improve use of whitespace
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:44 +01:00
Emilia Hane
b2abec5d35
Verify StoreConfig 2023-02-08 11:44:44 +01:00
Emilia Hane
00ca21e84c
Make implementation of BlobInfo more coder friendly 2023-02-08 11:44:43 +01:00
Emilia Hane
8f137df02e
fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:43 +01:00
Emilia Hane
a2eda76291
Correct comment 2023-02-08 11:44:43 +01:00
Emilia Hane
9ee9b6df76
Remove unused stuff 2023-02-08 11:44:42 +01:00
Emilia Hane
6dff69bde9
Atomically update blob info with pruned blobs 2023-02-08 11:44:42 +01:00
Emilia Hane
5d2480c762
Improve naming 2023-02-08 11:44:42 +01:00
Emilia Hane
43c3c74a48
fixup! Fix blobs store bug 2023-02-08 11:44:41 +01:00
Emilia Hane
c50f83116e
Fix wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
f6346f89c1
Clarify comment
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:41 +01:00
Emilia Hane
e4b447395a
Clarify wording
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
756c881857
Keep uniform size small keys
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:40 +01:00
Emilia Hane
4de523fb75
fixup! Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
1812301c9c
Allow user to set an epoch margin for pruning 2023-02-08 11:44:40 +01:00
Emilia Hane
d7fc24a9d5
Plug in running blob pruning in migrator, related bug fixes and add todos 2023-02-08 11:44:40 +01:00
Emilia Hane
a875bec5f2
Fix blobs store bug 2023-02-08 11:44:39 +01:00
Emilia Hane
3bede06c9b
Fix typo 2023-02-08 11:44:38 +01:00
Emilia Hane
54699f808c
fixup! Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
83a9520761
Clarify hybrid blob prune solution and fix error handling 2023-02-08 11:44:38 +01:00
Emilia Hane
3d93dad0e2
Fix type bug
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-02-08 11:44:37 +01:00
Emilia Hane
44ec331452
fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
20567750c1
fixup! Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
7103a257ce
Simplify conceptual design 2023-02-08 11:44:37 +01:00
Emilia Hane
0d13932663
Fix epoch constructor misconception 2023-02-08 11:44:37 +01:00
Emilia Hane
b5abfe620a
Convert epochs_per_blob_prune to Epoch once 2023-02-08 11:44:36 +01:00
Emilia Hane
d58a30b3de
fixup! Store orphan block roots 2023-02-08 11:44:36 +01:00
Emilia Hane
6346c30158
Enable skipping blob pruning at each epoch 2023-02-08 11:44:35 +01:00
Emilia Hane
8752deeced
Store orphan block roots 2023-02-08 11:44:35 +01:00
Emilia Hane
94aa2cef67
Log info loaded from disk 2023-02-08 11:44:34 +01:00
Emilia Hane
a2b8c6ee69
Save fetching state for blobs pruning 2023-02-08 11:44:33 +01:00
Emilia Hane
6f5ca02ac9
Improve syntax
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-02-08 11:44:33 +01:00
Emilia Hane
667cca5cf2
Fix try_prune_blobs to use state root 2023-02-08 11:44:33 +01:00
Emilia Hane
d67468d737
Prune blobs on migration in addition to start-up 2023-02-08 11:44:32 +01:00
Emilia Hane
ce2db355de
Fix rebase conflict 2023-02-08 11:44:32 +01:00
Emilia Hane
a211e6afee
Fix rebase conflict 2023-02-08 11:44:31 +01:00
Emilia Hane
28e1e635c3
Fix rebase conflict 2023-02-08 11:44:31 +01:00
Emilia Hane
d3b94d8617
fixup! Prune blobs before data availability breakpoint 2023-02-08 11:44:31 +01:00
Emilia Hane
934f3ab587
Remove inaccurate guess for db index 2023-02-08 11:44:31 +01:00