Commit Graph

73 Commits

Author SHA1 Message Date
Martin Holst Swende
0e7efd696b
core/rawdb, ethdb: introduce batched/atomic reads from ancients (#23566)
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
2021-10-25 16:24:27 +02:00
Martin Holst Swende
794c6133ef
core/rawdb: freezer batch write (#23462)
This change is a rewrite of the freezer code.

When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.

In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.

To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-07 12:31:17 +02:00
Martin Holst Swende
5f98020a21
core/rawdb: implement sequential reads in freezer_table (#23117)
* core/rawdb: implement sequential reads in freezer_table

* core/rawdb, ethdb: add sequential reader to db interface

* core/rawdb: lint nitpicks

* core/rawdb: fix some nitpicks

* core/rawdb: fix flaw with deferred reads not being performed

* core/rawdb: better documentation
2021-08-13 11:51:01 +03:00
Péter Szilágyi
1a83114c74
all: update author list and licenses 2019-07-22 12:17:27 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
rjl493456442
b6cac42e9f
core/rawdb: add file lock for freezer 2019-05-16 10:39:31 +03:00
Péter Szilágyi
006c21efc7
cmd, core, eth, les, node: chain freezer on top of db rework 2019-05-16 10:39:29 +03:00
Martin Holst Swende
59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
Péter Szilágyi
d6225ab846
cmd/utils, eth: relinquish GC cache to read cache in archive mode 2019-02-07 14:36:25 +02:00
Dmitrij Koniajev
6a7695e367 ethdb, rpc: support building on js/wasm (#17709)
The changes allow building WebAssembly applications which use ethclient.Client.
2018-10-16 00:47:25 +02:00
gary rong
5d1d1a808d consensus, ethdb, metrics: implement forced-meter (#17667) 2018-09-17 15:32:34 +03:00
gary rong
e8824f6e74 vendor, ethdb: resume write operation asap (#17144)
* vendor: update leveldb

* ethdb: remove useless warning log
2018-07-12 11:07:51 +03:00
gary rong
a4a2343cdc ethdb, core: implement delete for db batch (#17101) 2018-07-02 11:16:30 +03:00
gary rong
f991995918 ethdb: gracefullly handle quit channel (#16794)
* ethdb: gratefullly handle quit channel

* ethdb: minor polish
2018-06-11 16:11:48 +03:00
gary rong
6ce21a4744 vendor, ethdb: print warning log if leveldb is performing compaction (#16766)
* vendor: update leveldb package

* ethdb: print warning log if db is performing compaction

* ethdb: update annotation and log
2018-05-22 11:12:52 +03:00
gary rong
6b2b328cdb ethdb: add leveldb write delay statistic (#16499) 2018-04-16 11:18:24 +03:00
Felix Lange
0fac705ed0
compression/rle: delete RLE compression (#16468) 2018-04-09 13:47:06 +02:00
Péter Szilágyi
495bdb0c71
cmd: export preimages in RLP, support GZIP, uniform with block export 2018-03-26 14:08:01 +03:00
rjl493456442
b6b6f52ec8
cmd: implement preimage dump and import cmds 2018-03-26 12:51:46 +03:00
Kurkó Mihály
39c16c8a1e cmd, ethdb, vendor: integrate leveldb iostats (#16277)
* cmd, dashboard, ethdb, vendor: send iostats to dashboard

* ethdb: change names

* ethdb: handle parsing errors

* ethdb: handle iostats syntax error

* ethdb: r -> w
2018-03-08 14:59:00 +02:00
Anton Evangelatov
ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi
3e89b80ccb
ethdb: reset the batch size too on reset 2018-01-30 19:12:28 +02:00
Martin Holst Swende
017b9f7eac core, ethdb: reuse database batches (#15989)
* leveldb: Update leveldb to 211f780 (poolfix)

* core, ethdb: reuse database batches
2018-01-30 19:03:31 +02:00
Felix Lange
10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
Péter Szilágyi
9184249b39 Logger updates 3 (#3730)
* accounts, cmd, eth, ethdb: port logs over to new system

* ethdb: drop concept of cache distribution between dbs

* eth: fix some log nitpicks to make them nicer
2017-03-02 14:06:16 +01:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Nick Johnson
d30d7800e0 ethdb: Implement interface for prefixed operations to the DB (#3536) 2017-01-11 13:26:09 +01:00
zsfelfoldi
7db7109a5b cmd, eth: added light client and light server modes 2016-11-09 02:12:53 +01:00
Felix Lange
44bc2e80dd Merge pull request #2914 from fjl/node-coinhabit
cmd/utils, node: make datadir reusable for bzzd
2016-09-29 14:24:37 +02:00
Péter Szilágyi
710435b51b core, eth, trie: reuse trie journals in all our code 2016-09-28 11:27:31 +03:00
Felix Lange
52ede09b17 ethdb: add accessor for database directory 2016-09-16 15:24:31 +02:00
Felix Lange
84d11c19fd eth: remove dapp database remains 2016-08-17 17:39:03 +02:00
villesundell
c7442ef0d2 ethdb, p2p/discover: replace "alloted" with "allotted" (#2785) 2016-07-12 11:42:47 +02:00
Péter Szilágyi
e90958cd29 cmd, eth, ethdb, node: prioritise chaindata for resources, bump cache 2016-03-09 10:33:39 +02:00
Péter Szilágyi
f1ddb1a7ad ethdb: fully disable metrics if not requested (oops?) 2016-02-23 15:47:04 +02:00
Felix Lange
b252589960 ethdb: remove Flush 2015-09-14 23:36:30 +02:00
Felix Lange
8b32f10f16 ethdb: add NewBatch 2015-09-14 23:36:30 +02:00
Jeffrey Wilcke
a23478c0be core, eth, trie, xeth: merged state, chain, extra databases in one 2015-08-07 22:29:02 +02:00
Jeffrey Wilcke
1fad8798ec Merge pull request #1515 from fjl/license-fixes
all: fix license headers one more time
2015-07-28 04:29:42 -07:00
Felix Lange
bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Jeffrey Wilcke
8f56eea77d ethdb, trie: removed RLE compression 2015-07-23 12:57:33 +02:00
Jeffrey Wilcke
0cff61beda Merge pull request #1510 from fjl/license-fixes
all: license fixes
2015-07-23 01:43:11 -07:00
Felix Lange
3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Péter Szilágyi
c7e7778f2a cmd, core, eth, ethdb: cache flag to allocate memory for db internal use 2015-07-22 14:00:52 +03:00
Péter Szilágyi
eb2d168710 eth, ethdb: fix a data race during startup/shutdown 2015-07-09 12:44:07 +03:00
Felix Lange
ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Péter Szilágyi
ccbb56b4f2 cmd/geth, eth, ethdb: monitor database compactions 2015-06-30 00:51:29 +02:00
obscuren
2a5a55efaf ethdb: accessor for LDB. TODO remove this interface 2015-06-29 18:51:48 +02:00