Commit Graph

73 Commits

Author SHA1 Message Date
ucwong
a425a47ddc
core/rawdb, eth/protocols : Method name typo fix (#22026) 2020-12-27 22:38:16 +01:00
Péter Szilágyi
017831dd5b
core, eth: split eth package, implement snap protocol (#21482)
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md) , but does not enable it by default. 

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende
4d48980e74
core, eth, les: implement unclean-shutdown marker (#21893)
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again. 
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems. 

Co-authored-by: Nagy Salem <me@muhnagy.com>
2020-12-11 15:56:00 +01:00
gary rong
b63e3c37a6
core: improve snapshot journal recovery (#21594)
* core/state/snapshot: introduce snapshot journal version

* core: update the disk layer in an atomic way

* core: persist the disk layer generator periodically

* core/state/snapshot: improve logging

* core/state/snapshot: forcibly ensure the legacy snapshot is matched

* core/state/snapshot: add debug logs

* core, tests: fix tests and special recovery case

* core: polish

* core: add more blockchain tests for snapshot recovery

* core/state: fix comment

* core: add recovery flag for snapshot

* core: add restart after start-after-crash tests

* core/rawdb: fix imports

* core: fix tests

* core: remove log

* core/state/snapshot: fix snapshot

* core: avoid callbacks in SetHead

* core: fix setHead cornercase where the threshold root has state

* core: small docs for the test cases

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-10-29 21:01:58 +02:00
gary rong
9d25f34263
core: track and improve tx indexing/unindexing (#21331)
* core: add background indexer to waitgroup

* core: make indexer stopable

* core/rawdb: add unit tests

* core/rawdb: fix lint

* core/rawdb: fix tests

* core/rawdb: fix linter
2020-10-20 16:34:50 +02:00
Giuseppe Bertone
07751c3d26
cmd/geth: added counters to the geth inspect report (#21495)
* database: added counters

* Improved stats for ancient db

* Small improvement

* Better message and added percentage while counting receipts

* Fast counting for receipts

* added info message

* Show both receips itemscount  from ancient db and counted receipts

* Fixed default case

* Removed counter for receipts in ancient store

* Removed counting of receipts present in leveldb
2020-09-17 10:23:56 +02:00
Giuseppe Bertone
0185ee0993
core/rawdb: single point of maintenance for writing and deleting tx lookup indexes (#21480) 2020-09-15 10:37:01 +02:00
Péter Szilágyi
5655dce3b8
core/rawdb: only complain loudly if truncating many items 2020-08-25 09:03:14 +03:00
gary rong
87c0ba9213
core, eth, les, trie: add a prefix to contract code (#21080) 2020-08-21 15:10:40 +03:00
Péter Szilágyi
8cbdc8638f
core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
2020-08-20 13:01:24 +03:00
rene
c0c01612e9
node: refactor package node (#21105)
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.

There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.

There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.

The --graphql.port and --graphql.addr flags are no longer available.
2020-08-03 19:40:46 +02:00
meowsbits
ff90894636
core/rawdb: convert some comments to godoc convention (#21384) 2020-07-29 21:53:59 +02:00
gary rong
79addac698
core/rawdb: better log messages for ancient failure (#21327) 2020-07-13 20:43:30 +02:00
gary rong
6eef141aef
les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
2020-07-13 11:02:54 +02:00
AusIV
3ebfeb09fe
core/rawdb: fix high memory usage in freezer (#21243)
The ancients variable in the freezer is a list of hashes, which
identifies all of the hashes to be frozen. The slice is being allocated
with a capacity of `limit`, which is the number of the last block
this batch will attempt to add to the freezer. That means we are
allocating memory for all of the blocks in the freezer, not just
the ones to be added.

If instead we allocate `limit - f.frozen`, we will only allocate
enough space for the blocks we're about to add to the freezer. On
mainnet this reduces usage by about 320 MB.
2020-06-19 10:51:37 +03:00
zhangsoledad
bcf19bc4be
core/rawdb: swap tailId and itemOffset for deleted items in freezer (#21220)
* fix(freezer): tailId filenum offset were misplaced

* core/rawdb: assume first item in freezer always start from zero
2020-06-17 09:41:07 +02:00
Martin Michlmayr
4f2784b38f
all: fix typos in comments (#21118) 2020-05-25 10:21:28 +02:00
ucwong
7540c53e72
core/rawdb: remove unused math (#21065) 2020-05-12 12:19:15 +03:00
ucwong
aaede53738
core/rawdb : log format fix for Unindexing transaction (#21064)
* core/rawdb : log format fix for Unindexing transaction

* core/rawdb: tiny fixup

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-12 11:46:35 +03:00
Martin Holst Swende
4535230059
cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-11 18:58:43 +03:00
AusIV
069a7e1f8a
core/rawdb: stop freezer process as part of freezer.Close() (#21010)
* core/rawdb: Stop freezer process as part of freezer.Close()

When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.

This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.

Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.

* core/rawdb: Fix formatting for golint

* core/rawdb: Use backoff flag to avoid repeating select

* core/rawdb: Include accidentally omitted backoff
2020-05-11 15:11:17 +03:00
Martin Holst Swende
6402c42b67
all: simplify and fix database iteration with prefix/start (#20808)
* core/state/snapshot: start fixing disk iterator seek

* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start

* les, core/state/snapshot: iterator fixes

* all: remove two iterator methods

* all: rename Iteratee.NewIteratorWith -> NewIterator

* ethdb: fix review concerns
2020-04-15 14:08:53 +03:00
Marius van der Wijden
2a836bb259
core/rawdb: fix data race between Retrieve and Close (#20919)
* core/rawdb: fixed data race between retrieve and close

closes https://github.com/ethereum/go-ethereum/issues/20420

* core/rawdb: use non-atomic load while holding mutex
2020-04-14 18:13:47 +03:00
Péter Szilágyi
613af7ceea
Merge pull request #20152 from karalabe/snapshot-5
Dynamic state snapshots
2020-03-23 12:57:31 +02:00
Martin Holst Swende
074efe6c8d
core: fix two snapshot iterator flaws, decollide snap storage prefix
* core/state/snapshot/iterator: fix two disk iterator flaws

* core/rawdb: change SnapshotStoragePrefix to avoid prefix collision with preimagePrefix
2020-03-23 12:34:27 +02:00
Péter Szilágyi
36e93d2dd8
Merge pull request #20779 from meowsbits/patch-3
core/rawdb: fix freezer table test error check
2020-03-18 16:30:58 +02:00
gary rong
6283391c99
core/rawdb: improve table database (#20703)
This PR fixes issues in TableDatabase.

TableDatabase is a wrapper of underlying ethdb.Database with an additional prefix.
The prefix is applied to all entries it maintains. However when we try to retrieve entries
from it we don't handle the key properly. In theory the prefix should be truncated and
only user key is returned. But we don't do it in some cases, e.g. the iterator and batch
replayer created from it. So this PR is the fix to these issues.
2020-03-18 13:15:49 +01:00
meowsbits
20a092fb9f
core/rawdb: fix freezer table test error check
Fixes: Condition is always 'false' because 'err' is always 'nil'
2020-03-18 06:55:30 -05:00
Péter Szilágyi
fd39f722a3
core: journal the snapshot inside leveldb, not a flat file 2020-02-25 12:51:10 +02:00
Péter Szilágyi
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation 2020-02-25 12:51:08 +02:00
Péter Szilágyi
d754091a87
core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup 2020-02-25 12:51:07 +02:00
Péter Szilágyi
542df8898e
core: initial version of state snapshots 2020-02-25 12:51:04 +02:00
Péter Szilágyi
58e33d9e5a
Merge pull request #20403 from karalabe/fix-freezer-reinit
core/rawdb: fix reinit regression caused by the hash check PR
2019-11-27 15:05:58 +02:00
Péter Szilágyi
717f8a4e8f
core/rawdb: fix reinit regression caused by the hash check PR 2019-11-27 14:41:47 +02:00
Guillaume Ballet
7b189d6f1f
core: fix staticcheck warnings (#20384)
* core: fix staticcheck warnings

* fix goimports
2019-11-27 09:50:30 +01:00
gary rong
b9c90c5581 core/rawdb: check hash before return data from ancient db (#20195)
* core/rawdb: check hash before return data from ancient db

* core/rawdb: fix lint

* core/rawdb: calculate the hash in the fly
2019-11-19 12:32:57 +02:00
Felix Lange
689486449d build: use golangci-lint (#20295)
* build: use golangci-lint

This changes build/ci.go to download and run golangci-lint instead
of gometalinter.

* core/state: fix unnecessary conversion

* p2p/simulations: fix lock copying (found by go vet)

* signer/core: fix unnecessary conversions

* crypto/ecies: remove unused function cmpPublic

* core/rawdb: remove unused function print

* core/state: remove unused function xTestFuzzCutter

* core/vm: disable TestWriteExpectedValues in a different way

* core/forkid: remove unused function checksum

* les: remove unused type proofsData

* cmd/utils: remove unused functions prefixedNames, prefixFor

* crypto/bn256: run goimports

* p2p/nat: fix goimports lint issue

* cmd/clef: avoid using unkeyed struct fields

* les: cancel context in testRequest

* rlp: delete unreachable code

* core: gofmt

* internal/build: simplify DownloadFile for Go 1.11 compatibility

* build: remove go test --short flag

* .travis.yml: disable build cache

* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement

* .golangci.yml: enable goconst and ineffassign linters

* build: print message when there are no lint issues

* internal/build: refactor download a bit
2019-11-18 10:49:17 +02:00
Péter Szilágyi
72d5a27a39
core, metrics, p2p: switch some invalid counters to gauges 2019-09-10 14:39:07 +03:00
Péter Szilágyi
1a83114c74
all: update author list and licenses 2019-07-22 12:17:27 +03:00
Christian Muehlhaeuser
57fc1d21e1 cmd/geth, core/rawdb: add missing error checks (#19871)
* Added missing error checks

Add error handling where we assign err a value, but don't check for it being nil.

* core/rawdb: tiny style nit
2019-07-22 10:51:13 +03:00
大彬
4ac04ae0fe all: replace fmt.Print* calls with t.Log* in tests (#19670) 2019-07-17 13:20:24 +02:00
Péter Szilágyi
b02958b9c5
core, ethdb, metrics, p2p: expose various counter metrics for grafana 2019-06-11 09:49:13 +03:00
Frank Szendzielarz
f9c0e093ed core/rawdb: avoid O_APPEND (#19676)
* Fix file system access for Windows

* Encapsulate file accesses

* Style fixes
2019-06-10 12:45:12 +03:00
Péter Szilágyi
fc85777a21
core: concurrent database reinit from freezer dump
* core: reinit chain from freezer in batches

* core/rawdb: concurrent database reinit from freezer dump

* core/rawdb: reinit from freezer in sequential order
2019-05-27 15:48:30 +03:00
Péter Szilágyi
7392f59e7c
core/rawdb: keep genesis in key-value store for full sync too 2019-05-27 12:25:24 +03:00
Péter Szilágyi
f35975ea21
core/rawdb, eth/downloader: align 64bit atomic fields 2019-05-17 01:45:05 +03:00
Péter Szilágyi
9eba3a9fff
cmd/geth, core/rawdb: seamless freezer consistency, friendly removedb 2019-05-16 17:01:56 +03:00
Péter Szilágyi
536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
Martin Holst Swende
42c746d6f4
freezer: disable compression on hashes and difficulties (#14)
* freezer: disable compression on hashes and difficulties

* core/rawdb: address review concerns

* core/rawdb: address review concerns
2019-05-16 10:39:33 +03:00