Commit Graph

93 Commits

Author SHA1 Message Date
Matthew Halpern
fbedf62f3d swarm/storage: fix loop bound for database cleanup (#19085)
The current loop continuation condition is always true as a uint8
is always being checked whether it is less than 255 (its maximum
value). Since the loop starts with the value 1, the loop termination
can be guarranteed to exit once the value overflows to 0.
2019-02-21 06:37:32 +01:00
Ferenc Szabo
e38b227ce6 Ci race detector handle failing tests (#19143)
* swarm/storage: increase mget timeout in common_test.go

 TestDbStoreCorrect_1k sometimes timed out with -race on Travis.

--- FAIL: TestDbStoreCorrect_1k (24.63s)
    common_test.go:194: testStore failed: timed out after 10s

* swarm: remove unused vars from TestSnapshotSyncWithServer

nodeCount and chunkCount is returned from setupSim and those values
we use.

* swarm: move race/norace helpers from stream to testutil

As we will need to use the flag in other packages, too.

* swarm: refactor TestSwarmNetwork case

Extract long running test cases for better visibility.

* swarm/network: skip TestSyncingViaGlobalSync with -race

As panics on Travis.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7e351b]

* swarm: run TestSwarmNetwork with fewer nodes with -race

As otherwise we always get test failure with `network_test.go:374:
context deadline exceeded` even with raised `Timeout`.

* swarm/network: run TestDeliveryFromNodes with fewer nodes with -race

Test on Travis times out with 8 or more nodes if -race flag is present.

* swarm/network: smaller node count for discovery tests with -race

TestDiscoveryPersistenceSimulationSimAdapters failed on Travis with
`-race` flag present. The failure was due to extensive memory usage,
coming from the CGO runtime. Using a smaller node count resolves the
issue.

=== RUN   TestDiscoveryPersistenceSimulationSimAdapter
==7227==ERROR: ThreadSanitizer failed to allocate 0x80000 (524288) bytes of clock allocator (error code: 12)
FATAL: ThreadSanitizer CHECK failed: ./gotsan.cc:6976 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
FAIL    github.com/ethereum/go-ethereum/swarm/network/simulations/discovery     804.826s

* swarm/network: run TestFileRetrieval with fewer nodes with -race

Otherwise we get a failure due to extensive memory usage, as the CGO
runtime cannot allocate more bytes.

=== RUN   TestFileRetrieval
==7366==ERROR: ThreadSanitizer failed to allocate 0x80000 (524288) bytes of clock allocator (error code: 12)
FATAL: ThreadSanitizer CHECK failed: ./gotsan.cc:6976 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
FAIL	github.com/ethereum/go-ethereum/swarm/network/stream	155.165s

* swarm/network: run TestRetrieval with fewer nodes with -race

Otherwise we get a failure due to extensive memory usage, as the CGO
runtime cannot allocate more bytes ("ThreadSanitizer failed to
allocate").

* swarm/network: skip flaky TestGetSubscriptionsRPC on Travis w/ -race

Test fails a lot with something like:
 streamer_test.go:1332: Real subscriptions and expected amount don't match; real: 0, expected: 20

* swarm/storage: skip TestDB_SubscribePull* tests on Travis w/ -race

Travis just hangs...

ok  	github.com/ethereum/go-ethereum/swarm/storage/feed/lookup	1.307s
keepalive
keepalive
keepalive

or panics after a while.

Without these tests the race detector job is now stable. Let's
invetigate these tests in a separate issue:
https://github.com/ethersphere/go-ethereum/issues/1245
2019-02-20 22:57:42 +01:00
lash
d36e974ba3 swarm/network: Keep span across roundtrip (#19140)
* swarm/newtork: WIP Span request span until delivery and put

* swarm/storage: Introduce new trace across single fetcher lifespan

* swarm/network: Put span ids for sendpriority in context value

* swarm: Add global span store in tracing

* swarm/tracing: Add context key constants

* swarm/tracing: Add comments

* swarm/storage: Remove redundant fix for filestore

* swarm/tracing: Elaborate constants comments

* swarm/network, swarm/storage, swarm:tracing: Minor cleanup
2019-02-20 14:50:37 +01:00
Elad
5b8ae7885e swarm/storage: fix influxdb gc metrics report (#19102) 2019-02-15 07:41:42 +01:00
Elad
3ee09ba035 swarm/storage/netstore: add fetcher cancellation on shutdown (#19049)
swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run
2019-02-14 07:51:57 +01:00
Janoš Guljaš
3fd6db2bf6 swarm: fix network/stream data races (#19051)
* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461775049

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461768477

* swarm/network/stream: correctly use Simulation.Run callback

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461783804

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462273444

* p2p/simulations: fix a deadlock calling getRandomNode with lock

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462317407

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it
2019-02-13 13:03:23 +01:00
holisticode
3d22a46c94 swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (#19028)
* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice
2019-02-13 00:17:44 +01:00
Ferenc Szabo
27e3f96819 swarm: CI race detector test adjustments (#19017) 2019-02-08 17:07:11 +01:00
lash
0c10d37606 swarm/network, swarm/storage: Preserve opentracing contexts (#19022) 2019-02-08 16:57:48 +01:00
Janoš Guljaš
4f3d22f06c swarm/storage/localstore: new localstore package (#19015) 2019-02-07 18:40:26 +01:00
holisticode
41597c2856 swarm: Debug API and HasChunks() API endpoint (#18980) 2019-02-07 15:49:19 +01:00
holisticode
7f55b0cbd8 cmd/swarm: hashes command (#19008) 2019-02-07 13:51:24 +01:00
holisticode
3eff652a7b swarm/storage: Get all chunk references for a given file (#19002) 2019-02-06 12:16:43 +01:00
Ferenc Szabo
1c3aa8d9b1 swarm/storage: fix test timeout with -race by increasing mget timeout 2019-02-05 14:34:34 +01:00
Ferenc Szabo
3591fc603f swarm/storage: Fix race in TestLDBStoreCollectGarbage. Disable testLDBStoreRemoveThenCollectGarbage (#18512) 2019-01-24 12:34:12 +01:00
Anton Evangelatov
bbd120354a
swarm: bootnode-mode, new bootnodes and no p2p package discovery (#18498) 2019-01-24 12:02:18 +01:00
Janoš Guljaš
f728837ee6 swarm/storage: fix mockNetFetcher data races (#18462)
fixes: ethersphere/go-ethereum#1117
2019-01-16 14:31:32 +01:00
Janoš Guljaš
d70c4faf20 swarm: Fix T.Fatal inside a goroutine in tests (#18409)
* swarm/storage: fix T.Fatal inside a goroutine

* swarm/network/simulation: fix T.Fatal inside a goroutine

* swarm/network/stream: fix T.Fatal inside a goroutine

* swarm/network/simulation: consistent failures in TestPeerEventsTimeout

* swarm/network/simulation: rename sendRunSignal to triggerSimulationRun
2019-01-09 07:05:55 +01:00
Janoš Guljaš
56a3f6c03c swarm/storage/mock/test: fix T.Fatal inside a goroutine (#18399) 2019-01-07 14:32:01 +01:00
Dave McGregor
33d233d3e1
vendor, crypto, swarm: switch over to upstream sha3 package 2019-01-04 09:26:07 +02:00
Janoš Guljaš
1d284c201d swarm/storage: change Proximity function and add TestProximity test (#18379) 2019-01-03 06:17:59 +01:00
Anton Evangelatov
9e9fc87e70
swarm: remove unused/dead code (#18351) 2018-12-23 17:31:32 +01:00
Javier Peletier
fe86a707d8 swarm/storage: remove unused methods from Chunk interface (#18283) 2018-12-18 15:25:02 +01:00
Anton Evangelatov
d322c9d550
swarm/storage/feed: remove unused code (#18324) 2018-12-17 11:32:55 +01:00
Javier Peletier
4e6f53ac33 swarm/storage: simplify ChunkValidator interface (#18285) 2018-12-12 16:22:17 +01:00
Elad
bb724080ca cmd/swarm, metrics, swarm/api/client, swarm/storage, swarm/metrics, swarm/api/http: add instrumentation (#18274) 2018-12-11 09:21:58 +01:00
Janoš Guljaš
c207edf2a3 swarm: add database abstractions (shed package) (#18183) 2018-11-26 18:49:01 +01:00
Ferenc Szabo
d876f214e5 swarm/storage: move 'running migrations for' log line (#18120)
So that we only see the log message when we actually have to migrate.
2018-11-20 08:30:38 +01:00
Anton Evangelatov
3333fe660f
swarm/storage: speed up garbage collection and rpc tests (#18128) 2018-11-19 12:26:45 +01:00
lash
a6942b9f25 swarm/storage: Batched database migration (#18113) 2018-11-15 14:57:03 +01:00
Alexey Sharov
eb8fa3cc89 cmd/swarm, swarm/api/http, swarm/bmt, swarm/fuse, swarm/network/stream, swarm/storage, swarm/storage/encryption, swarm/testutil: use pseudo-random instead of crypto-random for test files content generation (#18083)
- Replace "crypto/rand" to "math/rand" for files content generation
- Remove swarm/network_test.go.Shuffle and swarm/btm/btm_test.go.Shuffle - because go1.9 support dropped (see https://github.com/ethereum/go-ethereum/pull/17807 and comments to swarm/network_test.go.Shuffle)
2018-11-14 09:21:14 +01:00
Ferenc Szabo
c41e1bd1eb swarm/storage: fix garbage collector index skew (#18080)
On file access LDBStore's tryAccessIdx() function created a faulty
GC Index Data entry, because not indexing the ikey correctly.
That caused the chunk addresses/hashes to start with '00' and the last
two digits were dropped. => Incorrect chunk address.

Besides the fix, the commit also contains a schema change which will
run the CleanGCIndex() function to clean the GC index from erroneous
entries.

Note: CleanGCIndex() rebuilds the index from scratch which can take
a really-really long time with a huge DB (possibly an hour).
2018-11-13 15:22:53 +01:00
Ferenc Szabo
8080265f3f swarm/storage: fix access count on dbstore after cache hit (#17978)
Access count was not incremented when chunk was retrieved
from cache. So the garbage collector might have deleted the most
frequently accessed chunk from disk.

Co-authored-by: Ferenc Szabo <ferenc.szabo@ethereum.org>
2018-11-13 07:41:01 +01:00
Anton Evangelatov
cf3b187bde swarm, cmd/swarm: address ineffectual assignments (#18048)
* swarm, cmd/swarm: address ineffectual assignments

* swarm/network: remove unused vars from testHandshake

* swarm/storage/feed: revert cursor changes
2018-11-07 20:39:08 +01:00
Ferenc Szabo
54f650a3be swarm: clean up unused private types and functions (#17989)
* swarm: clean up unused private types and functions

Those that were identified by code inspection tool.

* swarm/storage: move/add Proximity GoDoc from deleted private function

The mentioned proximity() private function was deleted in:
1ca8fc1e6fa0ab4ab1aaca06d6fb32e173cd5f2f
2018-10-27 16:18:42 +02:00
lash
dc3c3fb1e1 swarm/storage: Add accessCnt for GC (#17845) 2018-10-12 16:25:38 +02:00
Anton Evangelatov
0fe9a372b3
swarm, swarm/storage: lower constants for faster tests (#17876)
* swarm/storage: lower constants for faster tests

* swarm: reduce test size for TestLocalStoreAndRetrieve

* swarm: reduce nodes for dec_inc_node_count
2018-10-09 11:45:42 +02:00
Javier Peletier
c5cb214f68 swarm/storage/feed: Expose MaxUpdateDataLength constant (#17858) 2018-10-08 10:57:38 +02:00
Anton Evangelatov
8c63d0d2e4 swarm/storage: extract isValid. correctly remove invalid chunks from store on migration (#17835) 2018-10-04 18:13:48 +03:00
Viktor Trón
e5677114dc
Merge pull request #17796 from epiclabs-io/mru-feeds
swarm/storage/feeds: Renamed MRU to Swarm Feeds
2018-10-03 14:59:41 +02:00
Anton Evangelatov
303b99663e
swarm: schemas and migrations (#17813) 2018-10-03 14:31:59 +02:00
Javier Peletier
de01178c18 swarm/storage/feed: Renamed package 2018-10-03 09:15:28 +02:00
Javier Peletier
696bc9b01c swarm/storage/feeds: renamed vars that can conflict with package name 2018-10-03 09:12:06 +02:00
Javier Peletier
58c0879c2f swarm/storage/feeds: removed capital Feed throughout 2018-10-03 09:12:06 +02:00
Javier Peletier
b6ccc06cda swarm/storage/feeds: Final package rename and moved files 2018-10-03 09:12:06 +02:00
Javier Peletier
83705ef6aa swarm/storage/mru: Renamed rest of MRU references 2018-10-03 09:12:06 +02:00
Javier Peletier
b35622cf3c swarm/storage/mru: Renamed all comments to Feeds 2018-10-03 09:12:06 +02:00
Javier Peletier
f1e86ad9cf swarm/storage/mru: Renamed all identifiers to Feeds 2018-10-03 09:12:06 +02:00
Janoš Guljaš
7e1c374dc6 swarm/storage: ensure 64bit hasherStore struct alignment (#17766) 2018-09-28 20:04:56 +02:00
Anton Evangelatov
0ee44e796a swarm/storage: make linter happy 2018-09-28 17:11:13 +02:00