roysc/plugeth
1
0
Fork 0
forked from cerc-io/plugeth
Code Pull Requests Activity
10,906 Commits 2 Branches 1 Tag 247 MiB
9b1543c282
Commit Graph

382 Commits

Author SHA1 Message Date
Elad
9b1543c282 swarm/network: update syncer metrics 2019-05-10 12:29:27 +02:00
Anton Evangelatov
297fa1855c swarm/network: measure addPeer and deletePeer to know if Kad rearranged 
swarm/storage: remove traces for put/get/set (#1389)

* swarm/storage: remove traces for put/get/set

* swarm/storage: remove Has traces
 2019-05-10 12:29:27 +02:00
Elad
84dfaea246 swarm: instrument setNextBatch 
swarm/storage/localstore: add gc metrics, disable flaky test
 2019-05-10 12:29:22 +02:00
Anton Evangelatov
3e9ba57669 swarm/storage: improve instrumentation 
swarm/storage/localstore: fix broken metric (#1373)

p2p/protocols: count different messages (#1374)

cmd/swarm: disable snapshot create test due to constant flakes (#1376)

swarm/network: remove redundant goroutine (#1377)
 2019-05-10 12:27:04 +02:00
lash
7f753461ca swarm/pss: disable failing handshake test 2019-05-10 12:26:59 +02:00
Anton Evangelatov
8802b9ce7f swarm-smoke: add syncDelay flag 
swarm/network: add want delay timer to syncing (#1367)

swarm/network: synchronise peer.close() (#1369)
 2019-05-10 12:26:55 +02:00
Elad
ad6c39012f swarm: push tags integration - request flow 
swarm/api: integrate tags to count chunks being split and stored
swarm/api/http: integrate tags in middleware for HTTP `POST` calls and assert chunks being calculated and counted correctly
swarm: remove deprecated and unused code, add swarm hash to DoneSplit signature, remove calls to the api client from the http package
 2019-05-10 12:26:52 +02:00
Janoš Guljaš
3030893a21 swarm/network: update syncing 2019-05-10 12:26:49 +02:00
Elad
12240baf61 swarm/chunk: add tags data type 
* swarm/chunk: add tags backend to chunk package
 2019-05-10 12:26:45 +02:00
Anton Evangelatov
f8eb8fe64c cmd/swarm-smoke: check if chunks are at most prox host 
swarm/network: measure how many chunks a node delivers (#1358)
 2019-05-10 12:26:41 +02:00
Elad
a1cd7e6e92 p2p/protocols, swarm/network: fix resource leak with p2p teardown 2019-05-10 12:26:37 +02:00
Janoš Guljaš
c1213bd00c swarm: LocalStore metrics 
* swarm/shed: remove metrics fields from DB struct

* swarm/schunk: add String methods to modes

* swarm/storage/localstore: add metrics and traces

* swarm/chunk: unknown modes without spaces in String methods

* swarm/storage/localstore: remove bin number from pull subscription metrics

* swarm/storage/localstore: add resetting time metrics and code improvements
 2019-05-10 12:26:33 +02:00
Anton Evangelatov
993b145f25 swarm/storage/localstore: fix export db.Put signature 
cmd/swarm/swarm-smoke: improve smoke tests (#1337)

swarm/network: remove dead code (#1339)

swarm/network: remove FetchStore and SyncChunkStore in favor of NetStore (#1342)
 2019-05-10 12:26:30 +02:00
Janoš Guljaš
996755c4a8 cmd/swarm, swarm: LocalStore storage integration 2019-05-10 12:26:26 +02:00
Janoš Guljaš
3873a7314d swarm/network: fix data races in TestInitialPeersMsg test (#19490) 
* swarm/network: fix data races in TestInitialPeersMsg test

* swarm/network: add Kademlia.Saturation method with lock

* swarm/network: add Hive.Peer method to safely retrieve a bzz peer

* swarm/network: remove duplicate comment

* p2p/testing: prevent goroutine leak in ProtocolTester

* swarm/network: fix data race in newBzzBaseTesterWithAddrs

* swarm/network: fix goroutone leaks in testInitialPeersMsg

* swarm/network: raise number of peer check attempts in testInitialPeersMsg

* swarm/network: use Hive.Peer in Hive.PeerInfo function

* swarm/network: reduce the scope of mutex lock in newBzzBaseTesterWithAddrs

* swarm/storage: disable TestCleanIndex with race detector
 2019-04-25 21:33:18 +02:00
gluk256
d9403690ec swarm/pss: Fix flaky TestProxNetwork (#19471) 2019-04-19 11:15:17 +02:00
Péter Szilágyi
78ec90717a
swarm/version: bump version due to Geth hotfix release 2019-04-17 15:50:41 +03:00
Javier Peletier
73fc65bb5b swarm/storage/feed: add context handling/cancellation to Swarm Feeds lookup, fix bad hint lookup bug (#19353) 
* swarm/storage/feed/lookup: Add context handling/forwarding

* swarm/storage/feed/lookup: Add test to catch bad hint

* swarm/storage/feed/lookup: Added context cancellation test
 2019-04-16 07:13:02 +02:00
Ferenc Szabo
26b50e3ebe cmd/swarm: fix resource leaks in tests (#19443) 
* swarm/api: fix file descriptor leak in NewTestSwarmServer

Swarm storage (localstore) was not closed. That resulted a
"too many open files" error if `TestClientUploadDownloadRawEncrypted`
was run with `-count 1000`.

* cmd/swarm: speed up StartNewNodes() by parallelization

Reduce cluster startup time from 13s to 7s.

* swarm/api: disable flaky TestClientUploadDownloadRawEncrypted with -race

* swarm/storage: disable flaky TestLDBStoreCollectGarbage (-race)

With race detection turned on the disabled cases often fail with:
"ldbstore_test.go:535: expected surplus chunk 150 to be missing, but got no error"

* cmd/swarm: fix process leak in TestACT and TestSwarmUp

Each test run we start 3 nodes, but we did not terminate them. So
those 3 nodes continued eating up 1.2GB (3.4GB with -race) after test
completion.

6b6c4d1c27 changed how we start clusters
to speed up tests. The changeset merged together test cases
and introduced a global cluster. But "forgot" about termination.

Let's get rid of "global cluster" so we have a clear owner of
termination (some time sacrifice), while leaving subtests to use the
same cluster.
 2019-04-11 12:44:15 +02:00
Péter Szilágyi
b7dd225179
swarm/version: bump Swarm due to Geth hotfix release 2019-04-10 16:13:35 +03:00
Péter Szilágyi
5fc5971438
swarm/version: bump version due to Geth-only hotfix release 2019-04-09 12:19:24 +03:00
Péter Szilágyi
5b947c5004
swarm/version: bump version due to Geth maintenance release 2019-04-08 16:08:22 +03:00
Viktor Trón
0529015091
swarm/network: hive bug: needed shallow peers are not sent to nodes beyond connection's proximity order (#19326) 
* swarm/network: fix hive bug not sending shallow peers

-  hive bug: needed shallow peers were not sent to nodes beyond connection's proximity order
- add extensive protocol exchange tests for initial subPeersMsg-peersMsg exchange
- modify bzzProtocolTester to allow pregenerated overlay addresses

* swarm/network: attempt to fix hive persistance test

* swarm/network: fix TestHiveStatePersistance (#1320)

* swarm/network: remove trace lines from the hive persistance test

* address PR review comments

* swarm/network: address PR comments on TestInitialPeersMsg

 * eliminate *testing.T argument from bzz/hive protocoltesters
 * add sorting (only runs in test code) on peersMsg payload
 * add random (0 to MaxPeersPerPO) peers for each po
 * add extra peers closer to pivot than control
 2019-04-02 09:15:16 +02:00
lash
2f5b6cb442 swarm/network: Use different privatekey for bzz overlay in sim (#19313) 
* cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter

* cmd/p2p/swarm: Remove comments + config.Enode nomarshal

* p2p/simulations: Remove superfluous error check

* p2p/simulation: Move init enode comment

* swarm, p2p/simulations, cmd/swarm: Use nodekey in binary record sign

* swarm/network, swarm/pss: Dervice bzzkey

* swarm/pss: Remove unused function

* swarm/network: Store swarm private key in simulation bucket

* swarm/pss: Shorten TextProxNetwork shortrunning test timeout

* swarm/pss: Increase prox test timeout

* swarm/pss: Increase timeout slightly on shortrunning proxtest

* swarm/network: Simplify bucket instantiation in servicectx func

* p2p/simulations: Tcpport -> udpport

* swarm/network, swarm/pss: Simplify + correct lock in servicefunc sim

* swarm/network: Cleanup after rebase on extract swarm enode new

* p2p/simulations, swarm/network: Make exec disc test pass

* swarm/network: Prune ye olde comment

* swarm/pss: Correct revised bzzkey method call

* swarm/network: Clarify comment about privatekey generation data

* swarm/pss: Fix syntax errors after rebase

* swarm/network: Rename misleadingly named method

(amend commit to trigger ci - attempt 5)
 2019-03-22 21:37:25 +01:00
gluk256
8d04154691 p2p/simulations: wait until all connections are recreated when uploading snapshot (#19312) 
* swarm/network/simulation: test cases refactored

* swarm/pss: minor refactoring

* swarm/simulation: UploadSnapshot updated

* swarm/network: style fix

* swarm/pss: bugfix
 2019-03-22 11:20:17 +01:00
lash
09924cbcaa cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter (#19309) 
* cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter

* cmd/p2p/swarm: Remove comments + config.Enode nomarshal

* p2p/simulations: Remove superfluous error check

* p2p/simulation: Move init enode comment

* swarm/api: Check error in config test

* swarm, p2p/simulations, cmd/swarm: Use nodekey in binary record sign

* cmd/swarm: Make nodekey available for swarm api config
 2019-03-22 05:55:47 +01:00
Anton Evangelatov
baded64d88
swarm/network: measure time of messages in priority queue (#19250) 2019-03-20 21:30:34 +01:00
gluk256
6e401792ce swarm/pss: negihbourhood addressing simulation tests (#19278) 
* swarm/pss: fixed bug in pss.process, test added

* swarm/pss: test case updated

* swarm/pss: WaitTillSnapshotRecreated() func added

* swarm/pss: snapshot test updated

* swarm/pss: WaitTillSnapshotLoaded() fixed

* swarm/pss: gofmt applied

* swarm/pss: refactoring, file renamed

* swarm/pss: input data fixed

* swarm/pss: race condition fixed

* swarm/pss: test timeout increased

* swarm/pss: eliminated the global variables

* swarm/pss: tests added

* swarm/pss: comments added

* swarm/pss: comment fixed

* swarm/pss: refactored according to review

* swarm/pss: style fix

* swarm/pss: increased timeout
 2019-03-16 08:39:38 +01:00
Janoš Guljaš
f180981273 swarm/shed: add vector uint64 field (#19279) 2019-03-15 15:12:45 +01:00
lash
4b4f03ca37 swarm, p2p: Prerequities for ENR replacing handshake (#19275) 
* swarm/api, swarm/network, p2p/simulations: Prerequisites for handshake remove

* swarm, p2p: Add full sim node configs for protocoltester

* swarm/network: Make stream package pass tests

* swarm/network: Extract peer and addr types out of protocol file

* p2p, swarm: Make p2p/protocols tests pass + rename types.go

* swarm/network: Deactivate ExecAdapter test until binary ENR prep

* swarm/api: Remove comments

* swarm/network: Uncomment bootnode record load
 2019-03-15 11:27:17 +01:00
Anton Evangelatov
1a3e25e4c1
swarm: tracing improvements (#19249) 2019-03-11 11:45:34 +01:00
Janoš Guljaš
9a58a9b91a swarm/storage/localstore: global batch write lock (#19245) 
* swarm/storage/localstore: most basic database

* swarm/storage/localstore: fix typos and comments

* swarm/shed: add uint64 field Dec and DecInBatch methods

* swarm/storage/localstore: decrement size counter on ModeRemoval update

* swarm/storage/localstore: unexport modeAccess and modeRemoval

* swarm/storage/localstore: add WithRetrievalCompositeIndex

* swarm/storage/localstore: add TestModeSyncing

* swarm/storage/localstore: fix test name

* swarm/storage/localstore: add TestModeUpload

* swarm/storage/localstore: add TestModeRequest

* swarm/storage/localstore: add TestModeSynced

* swarm/storage/localstore: add TestModeAccess

* swarm/storage/localstore: add TestModeRemoval

* swarm/storage/localstore: add mock store option for chunk data

* swarm/storage/localstore: add TestDB_pullIndex

* swarm/storage/localstore: add TestDB_gcIndex

* swarm/storage/localstore: change how batches are written

* swarm/storage/localstore: add updateOnAccess function

* swarm/storage/localhost: add DB.gcSize

* swarm/storage/localstore: update comments

* swarm/storage/localstore: add BenchmarkNew

* swarm/storage/localstore: add retrieval tests benchmarks

* swarm/storage/localstore: accessors redesign

* swarm/storage/localstore: add semaphore for updateGC goroutine

* swarm/storage/localstore: implement basic garbage collection

* swarm/storage/localstore: optimize collectGarbage

* swarm/storage/localstore: add more garbage collection tests cases

* swarm/shed, swarm/storage/localstore: rename IndexItem to Item

* swarm/shed: add Index.CountFrom

* swarm/storage/localstore: persist gcSize

* swarm/storage/localstore: remove composite retrieval index

* swarm/shed: IterateWithPrefix and IterateWithPrefixFrom Index functions

* swarm/storage/localstore: writeGCSize function with leveldb batch

* swarm/storage/localstore: unexport modeSetRemove

* swarm/storage/localstore: update writeGCSizeWorker comment

* swarm/storage/localstore: add triggerGarbageCollection function

* swarm/storage/localstore: call writeGCSize on DB Close

* swarm/storage/localstore: additional comment in writeGCSizeWorker

* swarm/storage/localstore: add MetricsPrefix option

* swarm/storage/localstore: fix a typo

* swamr/shed: only one Index Iterate function

* swarm/storage/localstore: use shed Iterate function

* swarm/shed: pass a new byte slice copy to index decode functions

* swarm/storage/localstore: implement feed subscriptions

* swarm/storage/localstore: add more subscriptions tests

* swarm/storage/localsore: add parallel upload test

* swarm/storage/localstore: use storage.MaxPO in subscription tests

* swarm/storage/localstore: subscription of addresses instead chunks

* swarm/storage/localstore: lock item address in collectGarbage iterator

* swarm/storage/localstore: fix TestSubscribePull to include MaxPO

* swarm/storage/localstore: improve subscriptions

* swarm/storage/localstore: add TestDB_SubscribePull_sinceAndUntil test

* swarm/storage/localstore: adjust pull sync tests

* swarm/storage/localstore: remove writeGCSizeDelay and use literal

* swarm/storage/localstore: adjust subscriptions tests delays and comments

* swarm/storage/localstore: add godoc package overview

* swarm/storage/localstore: fix a typo

* swarm/storage/localstore: update package overview

* swarm/storage/localstore: remove repeated index change

* swarm/storage/localstore: rename ChunkInfo to ChunkDescriptor

* swarm/storage/localstore: add comment in collectGarbageWorker

* swarm/storage/localstore: replace atomics with mutexes for gcSize and tests

* swarm/storage/localstore: protect addrs map in pull subs tests

* swarm/storage/localstore: protect slices in push subs test

* swarm/storage/localstore: protect chunks in TestModePutUpload_parallel

* swarm/storage/localstore: fix a race in TestDB_updateGCSem defers

* swarm/storage/localstore: remove parallel flag from tests

* swarm/storage/localstore: fix a race in testDB_collectGarbageWorker

* swarm/storage/localstore: remove unused code

* swarm/storage/localstore: add more context to pull sub log messages

* swarm/storage/localstore: BenchmarkPutUpload and global lock option

* swarm/storage/localstore: pre-generate chunks in BenchmarkPutUpload

* swarm/storage/localstore: correct useGlobalLock in collectGarbage

* swarm/storage/localstore: fix typos and update comments

* swarm/storage/localstore: update writeGCSize comment

* swarm/storage/localstore: global batch write lock

* swarm/storage/localstore: remove global lock option

* swarm/storage/localstore: simplify DB.Close
 2019-03-09 00:06:39 +01:00
Anton Evangelatov
bb55b0fb53 swarm/storage: add comparison towards leveldb.ErrNotFound (#19243) 
* swarm/storage: add comparison towards leveldb.ErrNotFound

* swarm/storage: wrap leveldb ErrNotFound
 2019-03-08 17:28:57 +01:00
Anton Evangelatov
2cfe0bed9f swarm: fix relationship between spans in open tracing (#19236) 
* swarm/network: propagate span with ctx

* swarm/network: try to stop stream.send.request spans on time

* swarm/storage: add chunk ref as a log to netstore.fetcher span
 2019-03-08 08:52:25 +01:00
Janoš Guljaš
a6e5c6a2cc swarm/storage/localstore: fix synchronization in TestDB_gcSize (#19235) 2019-03-08 05:59:59 +01:00
Janoš Guljaš
eb199f1fc2 swarm: localstore hasser (#19230) 2019-03-07 10:07:54 +01:00
Ferenc Szabo
d45f8d1880 swarm/network: remove *WithServer tests from stream package (#19223) 
These tests never run as the build tag excluded them from the CI
execution. As a results the (dead) code got out of sync with other
parts of Swarm and now they would not even compile. => Removed.

resolves ethersphere/go-ethereum#1238
 2019-03-07 09:27:56 +01:00
holisticode
a87776a5fe swarm/network/stream: Fix flaky tests in GetSubscriptionsRPC test (#19227) 
* swarm/network/stream: fixed timing issues

* swarm/network/stream: only count first iteration of subscriptions

* swarm/network/stream/: fix linter errors
 2019-03-07 09:24:28 +01:00
holisticode
81ed700157 Enable longrunning tests to run (#19208) 
* p2p/simulations: increased snapshot load timeout for debugging

* swarm/network/stream: less nodes for snapshot longrunning tests

* swarm/network: fixed longrunning tests

* swarm/network/stream: store kademlia in bucket

* swarm/network/stream: disabled healthy check in delivery tests

* swarm/network/stream: longer SyncUpdateDelay for longrunning tests

* swarm/network/stream: more debug output

* swarm/network/stream: reduced longrunning snapshot tests to 64 nodes

* swarm/network/stream: don't WaitTillHealthy in SyncerSimulation

* swarm/network/stream: cleanup for PR
 2019-03-05 12:54:46 +01:00
Janoš Guljaš
216bd2ceba swarm/storage/localstore: fix testDB_collectGarbageWorker data race (#19206) 2019-03-04 22:19:57 +01:00
Anton Evangelatov
f9aa1cd21f Revert "swarm/network: Use actual remote peer ip in underlay (#19137)" (#19193) 
This reverts commit 460d206f30.
 2019-03-02 08:45:07 +01:00
Janoš Guljaš
b797dd07d2 swarm/shed, swarm/storage/localstore: add LastPullSubscriptionChunk (#19190) 
* swarm/shed, swarm/storage/localstore: add LastPullSubscriptionChunk

* swarm/shed: fix comments

* swarm/shed: fix TestIncByteSlice test

* swarm/storage/localstore: fix TestDB_LastPullSubscriptionChunk
 2019-03-02 08:44:22 +01:00
Anton Evangelatov
4e9230ea7a
swarm: enable p2p/discovery and disable dynamic dialling (#19189) 2019-03-01 12:20:37 +01:00
holisticode
994326ba00 swarm: new snapshot files (#19185) 2019-02-28 22:30:36 +01:00
lash
62d9d63858 swarm/network: WIP consider all nodes for healthy iteration (#19155) 
* swarm/network: WIP consider all nodes for healthy iteration

* swarm/network/simulation: extend TestWaitTillHealthy to really check kads are healthy

* cmd/swarm/swarm-snapshot: fixed bugs in snapshot creation binary

* swarm/network/simulation: addressed PR comments

* swarm/network/simulation: defer sim.Clsoe()

* swarm/network/simulation: fixed wrong sim.Close()

* swarm/network/simulation: addressed PR comments

* cmd/swarm/swarm-snapshot: reducing default to 8 nodes, more to 4

* cmd/swarm/swarm-snapshot: extended timeout to 3 mins, or 256 nodes snapshot times out

* swarm/network/simulation: More PR comments
 2019-02-28 08:12:50 +01:00
Janoš Guljaš
f0233948d2 swarm/chunk: move chunk related declarations to chunk package (#19170) 2019-02-26 16:09:32 +01:00
Janoš Guljaš
c83ba9e794 swarm/storage/localstore: fix tests for windows os (#19161) 2019-02-26 10:02:05 +01:00
Janoš Guljaš
340a53a98b swarm/pss: fix data race on HandshakeController.symKeyIndex (#19162) 
* swarm/pss: fix data race on HandshakeController.symKeyIndex

The HandshakeController.symKeyIndex map was accessed concurrently.
Since insufficient test coverage the race is not detected every time.
However, running TestClientHandshake a 100 times seems to be enough to
reproduce the race.

Note: I've chosen HandshakeController.lock to protect
HandshakeController.symKeyIndex as that was already protected in a few
functions by that lock.

Additionally:
- removed unused testStore
- enabled tests in handshake_test.go as they pass
- removed code duplication by adding getSymKey()

* swarm/pss: fix a data race on HandshakeController.keyC

* swarm/pss: fix data races with on Pss.symKeyPool
 2019-02-26 08:17:20 +01:00
Janoš Guljaš
872370e3bc swarm/network/simulation: do not copy node mutex in UploadSnapshot (#19160) 2019-02-25 10:03:31 +01:00
Matthew Halpern
81babe1509 swarm/*: remove redundant type specifiers (#19089) 2019-02-25 08:58:18 +01:00