Commit Graph

35 Commits

Author SHA1 Message Date
rjl493456442
e44d6551c3
cmd, core, ethdb, node: move chain freezer one folder deeper (#25487)
* cmd, core, ethdb, node: create chain freezer in a sub folder

* core/rawdb: remove unused code

* core, ethdb, node: add AncientDatadir API back

* cmd, core: extend freezer info dump for sub-ancient-store

* core/rawdb: rework freezer inspector

* core/rawdb: address comments from Peter

* core/rawdb: fix build issue
2022-08-08 12:08:36 +03:00
Marius van der Wijden
e6fa102eb0
core, eth, internal, rpc: implement final block (#24282)
* eth: core: implement finalized block

* eth/catalyst: fix final block

* eth/catalyst: update finalized head gauge

* internal/jsre/deps: updated web3.js to allow for finalized block

* eth/catalyst: make sure only one thread can call fcu

* eth/catalyst: nitpicks

* eth/catalyst: use plain mutex

* eth: nitpicks
2022-05-18 17:30:42 +03:00
rjl493456442
1941c5e6c9
core/rawdb: untie freezer and ancient chain data (#24684)
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.

This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
2022-05-06 13:28:42 +02:00
s7v7nislands
4024c1e869
fix typo (#24731) 2022-04-25 09:28:03 +02:00
rjl493456442
538a868384
core/rawdb, cmd, ethdb, eth: implement freezer tail deletion (#23954)
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion

* core/rawdb: address comments from martin and sina

* core/rawdb: fixes cornercase in tail deletion

* core/rawdb: separate metadata into a standalone file

* core/rawdb: remove unused code

* core/rawdb: add random test

* core/rawdb: polish code

* core/rawdb: fsync meta file before manipulating the index

* core/rawdb: fix typo

* core/rawdb: address comments
2022-03-10 09:37:23 +01:00
Martin Holst Swende
66a908c5e8
core/rawdb: fix double-lock causing hang (#24189)
Fixes #24159

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-01-04 19:02:37 +01:00
ucwong
adec878c1d
core/rawdb: fix error message fields in ReadLogs (#24104) 2021-12-14 11:11:05 +01:00
Martin Holst Swende
db03faa10d
core, eth: improve delivery speed on header requests (#23105)
This PR reduces the amount of work we do when answering header queries, e.g. when a peer
is syncing from us.

For some items, e.g block bodies, when we read the rlp-data from database, we plug it
directly into the response package. We didn't do that for headers, but instead read
headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it
in RLP-form as much as possible. When a node is syncing from us, it typically requests 192
contiguous headers. On master it has the following effect:

- For headers not in ancient: 2 db lookups. One for translating hash->number (even though
  the request is by number), and another for reading by hash (this latter one is sometimes
  cached).
  
- For headers in ancient: 1 file lookup/syscall for translating hash->number (even though
  the request is by number), and another for reading the header itself. After this, it
  also performes a hashing of the header, to ensure that the hash is what it expected. In
  this PR, I instead move the logic for "give me a sequence of blocks" into the lower
  layers, where the database can determine how and what to read from leveldb and/or
  ancients.

There are basically four types of requests; three of them are improved this way. The
fourth, by hash going backwards, is more tricky to optimize. However, since we know that
the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash
lookups.

The gapped collection can be optimized similarly, as a follow-up, at least in three out of
four cases.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-12-07 17:50:58 +01:00
Péter Szilágyi
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs (#23576)
* eth: request ID based message dispatcher

* eth: fix dispatcher cancellation, rework fetchers idleness tracker

* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00
Sina Mahmoodi
c57df9ca28
core/rawdb: add slow path for getting legacy logs (#23879)
* eth/tracers: add slow path for getting legacy logs

* core/rawdb: fix test
2021-11-11 15:04:06 +01:00
Martin Holst Swende
0e7efd696b
core/rawdb, ethdb: introduce batched/atomic reads from ancients (#23566)
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
2021-10-25 16:24:27 +02:00
Sina Mahmoodi
783e97ef1f
core/rawdb: avoid unnecessary receipt processing for log filtering (#23147)
* core/types: rm extranous check in test

* core/rawdb: add lightweight types for block logs

* core/rawdb,eth: use lightweight accessor for log filtering

* core/rawdb: add bench for decoding into rlpLogs
2021-09-28 12:54:49 +02:00
Martin Holst Swende
794c6133ef
core/rawdb: freezer batch write (#23462)
This change is a rewrite of the freezer code.

When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.

In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.

To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-07 12:31:17 +02:00
aaronbuchwald
56f533d00c
docs: fix docstring on read head block (#22776) 2021-04-29 19:23:07 +02:00
gary rong
0c70b83e00
all: add read-only option to database (#22407)
* all: add read-only option to database

* all: fixes tests

* cmd/geth: migrate flags

* cmd/geth: fix the compact

* cmd/geth: fix the format

* cmd/geth: fix log

* cmd: add chain-readonly

* core: add readonly notion to freezer

* core/rawdb: add log

* core/rawdb: fix freezer close

* cmd: fix

* cmd, core: construct db

* core: update tests
2021-03-22 19:06:30 +01:00
gary rong
5a1b384352
core: persist bad blocks (#21827)
* core: persist bad blocks

* core, eth, internal: address comments

* core/rawdb: add badblocks to inspector

* core, eth: update

* internal: revert

* core, eth: only save 10 bad blocks

* core/rawdb: address comments

* core/rawdb: fix

* core: address comments
2021-01-10 12:54:15 +01:00
Péter Szilágyi
8cbdc8638f
core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
2020-08-20 13:01:24 +03:00
gary rong
6eef141aef
les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
2020-07-13 11:02:54 +02:00
Martin Holst Swende
4535230059
cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-11 18:58:43 +03:00
Martin Holst Swende
6402c42b67
all: simplify and fix database iteration with prefix/start (#20808)
* core/state/snapshot: start fixing disk iterator seek

* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start

* les, core/state/snapshot: iterator fixes

* all: remove two iterator methods

* all: rename Iteratee.NewIteratorWith -> NewIterator

* ethdb: fix review concerns
2020-04-15 14:08:53 +03:00
gary rong
b9c90c5581 core/rawdb: check hash before return data from ancient db (#20195)
* core/rawdb: check hash before return data from ancient db

* core/rawdb: fix lint

* core/rawdb: calculate the hash in the fly
2019-11-19 12:32:57 +02:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Péter Szilágyi
006c21efc7
cmd, core, eth, les, node: chain freezer on top of db rework 2019-05-16 10:39:29 +03:00
Hongbin Mao
504f88b65b core/rawdb: typo fix storea => stores (#19498)
* typo fix

* change to stores
2019-04-26 12:22:21 +03:00
Péter Szilágyi
7221cb1434
core, eth, les, light: scope receipt functionality a bit cleaner 2019-04-15 13:42:26 +03:00
Matthew Halpern
6b0ddd141e
core, eth, les, light: store transaction receipts without txHash and gasCost 2019-04-15 13:15:39 +03:00
Brent
7fb89697fd core/types: add block location fields to receipt (#17662)
Solves #15210 without changing consensus, in a backwards compatible way,
by adding tx inclusion information to the Receipt struct.
2019-03-27 13:39:25 +01:00
Martin Holst Swende
59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
gary rong
7fd0ccaa68 core: remove unnecessary fields in logs, receipts and tx lookups (#17106)
* core: remove unnecessary fields in log

* core: bump blockchain database version

* core, les: remove unnecessary fields in txlookup

* eth: print db version explicitly

* core/rawdb: drop txlookup entry struct wrapper
2019-02-21 15:14:35 +02:00
Péter Szilágyi
accc0fab4f
core, eth/downloader: fix ancestor lookup for fast sync 2018-11-16 13:21:20 +02:00
Wuxiang
d98c45f70f core: fix a typo (#17941) 2018-10-19 16:33:27 +03:00
Wenbiao Zheng
aab7ab04b0 core/rawdb: wrap db key creations (#16914)
* core/rawdb: use wrappered helper to assemble key

* core/rawdb: wrappered helper to assemble key

* core/rawdb: rewrite the wrapper, pass common.Hash
2018-06-11 16:06:26 +03:00
Péter Szilágyi
6cf0ab38bd
core/rawdb: separate raw database access to own package (#16666) 2018-05-07 14:35:06 +03:00