Commit Graph

396 Commits

Author SHA1 Message Date
Péter Szilágyi
ff5538ad4c
params, swarm: begin Geth v1.8.18, Swarm v0.3.6 cycle 2018-10-09 10:37:49 +03:00
Péter Szilágyi
8bbe72075e
params, swarm: release Geth v1.8.17 and Swar v0.3.5 2018-10-09 10:35:31 +03:00
holisticode
11d0ff6578 Fix retrieval tests and simulation backends (#17723)
* swarm/network/stream: introduced visualized snapshot sync test

* swarm/network/stream: non-existing hash visualization sim

* swarm/network/stream: fixed retrieval tests; new backend for visualization

* swarm/network/stream: cleanup of visualized_snapshot_sync_sim_test.go

* swarm/network/stream: rebased PR on master

* swarm/network/stream: fixed loop logic in retrieval tests

* swarm/network/stream: fixed iterations for snapshot tests

* swarm/network/stream: address PR comments

* swarm/network/stream: addressed PR comments
2018-10-08 20:28:44 +02:00
Javier Peletier
c5cb214f68 swarm/storage/feed: Expose MaxUpdateDataLength constant (#17858) 2018-10-08 10:57:38 +02:00
Anton Evangelatov
8c63d0d2e4 swarm/storage: extract isValid. correctly remove invalid chunks from store on migration (#17835) 2018-10-04 18:13:48 +03:00
Viktor Trón
e5677114dc
Merge pull request #17796 from epiclabs-io/mru-feeds
swarm/storage/feeds: Renamed MRU to Swarm Feeds
2018-10-03 14:59:41 +02:00
Anton Evangelatov
303b99663e
swarm: schemas and migrations (#17813) 2018-10-03 14:31:59 +02:00
Javier Peletier
de01178c18 swarm/storage/feed: Renamed package 2018-10-03 09:15:28 +02:00
Javier Peletier
696bc9b01c swarm/storage/feeds: renamed vars that can conflict with package name 2018-10-03 09:12:06 +02:00
Javier Peletier
58c0879c2f swarm/storage/feeds: removed capital Feed throughout 2018-10-03 09:12:06 +02:00
Javier Peletier
68b8088cb9 swarm: Changed owners. 2018-10-03 09:12:06 +02:00
Javier Peletier
b6ccc06cda swarm/storage/feeds: Final package rename and moved files 2018-10-03 09:12:06 +02:00
Javier Peletier
83705ef6aa swarm/storage/mru: Renamed rest of MRU references 2018-10-03 09:12:06 +02:00
Javier Peletier
b35622cf3c swarm/storage/mru: Renamed all comments to Feeds 2018-10-03 09:12:06 +02:00
Javier Peletier
f1e86ad9cf swarm/storage/mru: Renamed all identifiers to Feeds 2018-10-03 09:12:06 +02:00
Alexey Sharov
dc5d643bb5 cmd/swarm, swarm: cross-platform Content-Type detection (#17782)
- Mime types generator (Standard "mime" package rely on system-settings, see mime.osInitMime)
- Changed swarm/api.Upload:
    - simplify I/O throttling by semaphore primitive and use file name where possible
    - f.Close() must be called in Defer - otherwise panic or future added early return will cause leak of file descriptors
    - one error was suppressed
2018-10-01 13:39:39 +02:00
Janoš Guljaš
7e1c374dc6 swarm/storage: ensure 64bit hasherStore struct alignment (#17766) 2018-09-28 20:04:56 +02:00
Anton Evangelatov
0ee44e796a swarm/storage: make linter happy 2018-09-28 17:11:13 +02:00
Javier Peletier
2c110c81ee Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification (#17559)
* swarm/storage/mru: Adaptive Frequency

swarm/storage/mru/lookup: fixed getBaseTime
Added NewEpoch constructor

swarm/api/client: better error handling in GetResource()


swarm/storage/mru: Renamed structures.
Renamed ResourceMetadata to ResourceID. 
Renamed ResourceID.Name to ResourceID.Topic

swarm/storage/mru: Added binarySerializer interface and test tools

swarm/storage/mru/lookup: Changed base time to time and + marshallers

swarm/storage/mru:  Added ResourceID (former resourceMetadata)

swarm/storage/mru: Added ResourceViewId and serialization tests

swarm/storage/mru/lookup: fixed epoch unmarshaller. Added Epoch Equals

swarm/storage/mru: Fixes as per review comments

cmd/swarm: reworded resource create/update help text regarding topic

swarm/storage/mru: Added UpdateLookup and serializer tests

swarm/storage/mru: Added UpdateHeader, serializers and tests

swarm/storage/mru: changed UpdateAddr / epoch to Base()

swarm/storage/mru: Added resourceUpdate serializer and tests

swarm/storage/mru: Added SignedResourceUpdate tests and serializers

swarm/storage/mru/lookup: fixed GetFirstEpoch bug

swarm/storage/mru: refactor, comments, cleanup

Also added tests for Topic
swarm/storage/mru: handler tests pass

swarm/storage/mru: all resource package tests pass

swarm/storage/mru: resource test pass after adding
timestamp checking support

swarm/storage/mru: Added JSON serializers to ResourceIDView structures

swarm/storage/mru: Sever, client, API test pass

swarm/storage/mru: server test pass

swarm/storage/mru: Added topic length check

swarm/storage/mru: removed some literals,
improved "previous lookup" test case

swarm/storage/mru: some fixes and comments as per review

swarm/storage/mru: first working version without metadata chunk

swarm/storage/mru: Various fixes as per review

swarm/storage/mru: client test pass

swarm/storage/mru: resource query strings and manifest-less queries


swarm/storage/mru: simplify naming

swarm/storage/mru: first autofreq working version



swarm/storage/mru: renamed ToValues to AppendValues

swarm/resource/mru: Added ToValues / FromValues for URL query strings

swarm/storage/mru: Changed POST resource to work with query strings.
No more JSON.

swarm/storage/mru: removed resourceid

swarm/storage/mru: Opened up structures

swarm/storage/mru: Merged Request and SignedResourceUpdate

swarm/storage/mru: removed initial data from CLI resource create

swarm/storage/mru: Refactor Topic as a direct fixed-length array

swarm/storage/mru/lookup: Comprehensive GetNextLevel tests

swarm/storage/mru: Added comments

Added length checks in Topic
swarm/storage/mru: fixes in tests and some code comments

swarm/storage/mru/lookup: new optimized lookup algorithm

swarm/api: moved getResourceView to api out of server

swarm/storage/mru: Lookup algorithm working

swarm/storage/mru: comments and renamed NewLookupParams

Deleted commented code


swarm/storage/mru/lookup: renamed Epoch.LaterThan to After

swarm/storage/mru/lookup: Comments and tidying naming



swarm/storage/mru: fix lookup algorithm

swarm/storage/mru: exposed lookup hint
removed updateheader

swarm/storage/mru/lookup: changed GetNextEpoch for initial values

swarm/storage/mru: resource tests pass

swarm/storage/mru: valueSerializer interface and tests



swarm/storage/mru/lookup: Comments, improvements, fixes, more tests

swarm/storage/mru: renamed UpdateLookup to ID, LookupParams to Query

swarm/storage/mru: renamed query receiver var



swarm/cmd: MRU CLI tests

* cmd/swarm: remove rogue fmt

* swarm/storage/mru: Add version / header for future use

* swarm/storage/mru: Fixes/comments as per review

cmd/swarm: remove rogue fmt

swarm/storage/mru: Add version / header for future use-

* swarm/storage/mru: fix linter errors

* cmd/swarm: Speeded up TestCLIResourceUpdate
2018-09-28 12:07:17 +02:00
Janos Guljas
a5e6bf7eef Merge branch 'master' into max-stream-peer-servers 2018-09-27 09:43:00 +02:00
Viktor Trón
c3cfdfacd0
Merge pull request #17757 from ethersphere/retrieve-request-ttl-pr
swarm: prevent forever running retrieve request loops
2018-09-27 07:08:58 +02:00
Viktor Trón
4b6824e07b
Merge pull request #17734 from frncmx/fix-dos-attack-invalid-hash-length
swarm/network/stream: fix DoS invalid offered hashes length
2018-09-26 12:44:42 +02:00
Balint Gabor
3f7acbbeb9 swarm: prevent forever running retrieve request loops 2018-09-26 11:34:40 +02:00
Janos Guljas
0d5e1e7bc9 swarm/network/stream: fix a typo in test comment 2018-09-26 11:30:45 +02:00
Alpay Yildirim
26cf866349 [ImgBot] optimizes images (#17741)
*Total -- 171.97kb -> 127.26kb (26%)

/swarm/api/testdata/test0/img/logo.png -- 17.71kb -> 4.02kb (77.29%)
/cmd/clef/sign_flow.png -- 35.54kb -> 20.27kb (42.98%)
/cmd/clef/docs/qubes/qrexec-example.png -- 18.66kb -> 15.79kb (15.4%)
/cmd/clef/docs/qubes/clef_qubes_http.png -- 13.97kb -> 11.95kb (14.44%)
/cmd/clef/docs/qubes/clef_qubes_qrexec.png -- 19.79kb -> 17.03kb (13.91%)
/cmd/clef/docs/qubes/qubes_newaccount-2.png -- 41.75kb -> 36.38kb (12.86%)
/cmd/clef/docs/qubes/qubes_newaccount-1.png -- 24.55kb -> 21.82kb (11.11%)
2018-09-26 10:39:39 +02:00
Janos Guljas
24349144b6 Merge branch 'master' into max-stream-peer-servers 2018-09-25 16:57:31 +02:00
Janos Guljas
7d56602391 swarm/api: fix TestDumpConfig 2018-09-25 16:32:11 +02:00
Felix Lange
30cd5c1854
all: new p2p node representation (#17643)
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.

Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.

The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.

* p2p/discover: port to p2p/enode

This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:

  - Table.Lookup is not available anymore. It used to take a public key
    as argument because v4 protocol requires one. Its replacement is
    LookupRandom.
  - Table.Resolve takes *enode.Node instead of NodeID. This is also for
    v4 protocol compatibility because nodes cannot be looked up by ID
    alone.
  - Types Node and NodeID are gone. Further commits in the series will be
    fixes all over the the codebase to deal with those removals.

* p2p: port to p2p/enode and discovery changes

This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.

New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.

* p2p/simulations, p2p/testing: port to p2p/enode

No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:

 - testing.ProtocolSession tracks complete nodes, not just their IDs.
 - adapters.NodeConfig has a new method to create a complete node.

These changes were needed to make swarm tests work.

Note that the NodeID change makes the code incompatible with old
simulation snapshots.

* whisper/whisperv5, whisper/whisperv6: port to p2p/enode

This port was easy because whisper uses []byte for node IDs and
URL strings in the API.

* eth: port to p2p/enode

Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.

* les: port to p2p/enode

Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.

* node: port to p2p/enode

This change simply replaces discover.Node and discover.NodeID with their
new equivalents.

* swarm/network: port to p2p/enode

Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).

There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.

Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
2018-09-25 00:59:00 +02:00
Janos Guljas
9e99a0c2b9 cmd/swarm, swarm: add stream peer servers limit 2018-09-24 17:56:00 +02:00
Péter Szilágyi
0ae462fb80
params, swarm: begin Geth v1.8.17, Swarm v0.3.5 cycle 2018-09-24 16:02:07 +03:00
Péter Szilágyi
477eb0933b
params, swarm: release Geth v1.8.16, Swarm v0.3.4 2018-09-24 15:59:21 +03:00
Ferenc Szabo
d3f056bd68 swarm/network/stream: fix DoS invalid hash length (#927) 2018-09-21 12:56:43 +02:00
HackyMiner
6975c72981 all: fix various comment typos (#17591)
* swarm: fixed comment typo
* eth: fixed comment typo
* cmd/puppeth: fixed comment typo
2018-09-19 18:10:40 +02:00
Balint Gabor
f5c7d1c8eb swarm/storage: Implement global timeout for fetcher (#17702) 2018-09-19 16:59:10 +02:00
Janoš Guljaš
1f45ba9bb1 swarm/network: downgrade fetcher unable to request log message severity (#17692) 2018-09-18 14:54:52 +02:00
Balint Gabor
bd58098f2d
swarm: Chunk refactor improvements (#17683)
* swarm/network: Protocol bump (for chunk refactor)

* swarm/network: Increase discovery and stream protocol version too

* swarm/network: Increase priority queue cap
2018-09-18 10:17:13 +02:00
Emil
86a03f97d3 all: simplify s[:] to s where s is a slice (#17673) 2018-09-14 22:07:13 +02:00
Balint Gabor
3ff2f75636
swarm: Chunk refactor (#17659)
Co-authored-by: Janos Guljas <janos@resenje.org>
Co-authored-by: Balint Gabor <balint.g@gmail.com>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>
Co-authored-by: Viktor Trón <viktor.tron@gmail.com>
2018-09-13 11:42:19 +02:00
Anton Evangelatov
ff3a5d24d2 swarm/storage: remove redundant increments for dataIdx and entryCnt (#17484)
* swarm/storage: remove redundant increments for dataIdx and entryCnt

* swarm/storage: add Delete to LDBStore

* swarm/storage: wait for garbage collection
2018-09-12 14:39:45 +02:00
Viktor Trón
bfce00385f Kademlia refactor (#17641)
* swarm/network: simplify kademlia/hive; rid interfaces

* swarm, swarm/network/stream, swarm/netork/simulations,, swarm/pss: adapt to new Kad API

* swarm/network: minor changes re review; add missing lock to NeighbourhoodDepthC
2018-09-12 11:24:56 +02:00
Viktor Trón
6dd87483d4 Encryption async api (#17603)
* swarm/storage/encryption: async segmentwise encryption/decryption

* swarm/storage: adapt hasherstore to encryption API change

* swarm/api: adapt RefEncryption for AC to new Encryption API

* swarm/storage/encryption: address review comments
2018-09-11 11:39:02 +02:00
Elad
8b9b149d54 swarm/api/http: bzz-immutable wrong handler bug (#17602) 2018-09-07 15:23:29 +02:00
Elad
70d31fb278 cmd/swarm: added password to ACT (#17598) 2018-09-07 09:56:05 +02:00
Elad
580145e96d swarm/storage: added metrics for db entry count (#17589) 2018-09-06 12:11:38 +02:00
Elad
4c15ffffdd swarm/api/http: added a regression test for resolver bug from #17483 (#17502) 2018-09-06 12:08:39 +02:00
Péter Szilágyi
62e94895da
params, swarm: begin geth v1.8.16 and swarm v0.3.4 cycle 2018-08-29 18:27:43 +03:00
Péter Szilágyi
89451f7c38
params, swarm: release geth v1.8.15 and swarm 0.3.3 2018-08-29 18:24:32 +03:00
Geon Kim
1acefafe22 swarm/api: fix typo (#17500) 2018-08-27 11:51:58 +03:00
Wenbiao Zheng
d1aa605f1e all: remove the duplicate 'the' in annotations (#17509) 2018-08-27 11:49:29 +03:00
Elad
f34f361ca6 swarm/api/http: fixed resolver bug (#17483) 2018-08-22 17:53:33 +03:00
Péter Szilágyi
2993fb519d
params, swarm: begin geth v1.8.15 and swarm v0.3.3 cycle 2018-08-22 11:42:03 +03:00
Péter Szilágyi
316fc7ecfc
params, swarm: release Geth v1.8.14 and Swarm v0.3.2 2018-08-22 11:39:00 +03:00
Geon Kim
522cfc68ff swarm: fix typos (#17473) 2018-08-21 21:13:33 +03:00
Anton Evangelatov
c582667c9b swarm/network: bump bzz protocol version (#17449) 2018-08-21 11:34:40 +03:00
Anton Evangelatov
dcd97c41fa swarm, swarm/network, swarm/pss: log error and fix logs (#17410)
* swarm, swarm/network, swarm/pss: log error and fix logs

* swarm/pss: log compressed publickey
2018-08-21 11:34:10 +03:00
Anton Evangelatov
a8aa89accb swarm/storage: cleanup task - remove bigger chunks (#17424) 2018-08-20 15:10:30 +03:00
Elad
c4078fc805 cmd/swarm: added swarm bootnodes (#17414) 2018-08-20 15:09:50 +03:00
Elad
e8752f4e9f cmd/swarm, swarm: added access control functionality (#17404)
Co-authored-by: Janos Guljas <janos@resenje.org>
Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>
Co-authored-by: Balint Gabor <balint.g@gmail.com>
2018-08-15 17:41:52 +02:00
Anton Evangelatov
97887d98da swarm/network, swarm/storage: validate chunk size (#17397)
* swarm/network, swarm/storage: validate default chunk size

* swarm/bmt, swarm/network, swarm/storage: update BMT hash initialisation

* swarm/bmt: move segmentCount to tests

* swarm/chunk: change chunk.DefaultSize to be untyped const

* swarm/storage: add size validator

* swarm/storage: add chunk size validation to localstore

* swarm/storage: move validation from localstore to validator

* swarm/storage: global chunk rules in MRU
2018-08-14 16:03:56 +02:00
Janoš Guljaš
6d1e292eef Manifest cli fix and upload defaultpath only once (#17375)
* cmd/swarm: fix manifest subcommands and add tests

* cmd/swarm: manifest update: update default entry for non-encrypted uploads

* swarm/api: upload defaultpath file only once

* swarm/api/client: improve UploadDirectory default path handling

* cmd/swarm: support absolute and relative default path values

* cmd/swarm: fix a typo in test

* cmd/swarm: check encrypted uploads in manifest update tests
2018-08-10 16:12:55 +02:00
Elad
3ec5dda4d2 swarm/api/http: added logging to denote request ended (#17371) 2018-08-10 13:49:37 +02:00
Janoš Guljaš
3bcb501c8f swarm/api: close tar writer in GetDirectoryTar to flush and clean (#17339) 2018-08-09 16:15:07 +02:00
Giulio M
a1eb9c7d13 swarm/api/http: fixed list leaf links (#17342) 2018-08-08 09:33:06 +02:00
Janoš Guljaš
00e6da9704 swarm/bmt: ignore data longer then 4096 bytes in Hasher.Write (#17338) 2018-08-07 15:34:33 +02:00
Attila Gazso
9df16f3468 swarm: Added lightnode flag (#17291)
* swarm: Added lightnode flag

Added --lightnode command line parameter
Added LightNode to Handshake message

* swarm/config: Fixed variable naming

* cmd/swarm: Changed BoolTFlag to BoolFlag for SwarmLightNodeEnabled

* swarm/network: Changed logging

* swarm/network: Changed protocol version testing

* swarm/network: Renamed DefaultNetworkID variable to TestProtocolNetworkID

* swarm/network: Bumped protocol version

* swarm/network: Changed LightNode handhsake test to table driven

* swarm/network: Changed back TestProtocolVersion to 5 for now

* swarm/network: Moved the test configuration inside the test function scope
2018-08-07 15:34:11 +02:00
Elad
4bb2dc3d09 swarm/api/http: test fixes (#17334) 2018-08-07 13:40:38 +02:00
Oleg Kovalov
cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
Anton Evangelatov
de9b0660ac swarm/README: add more sections to easily onboard developers (#17333) 2018-08-07 12:56:01 +02:00
Andrew Chiw
042191338d swarm/api/http: GET/PUT/PATCH/DELETE/POST multipart form unit tests. (#17277)
httpDo has a verbose option that dumps the HTTP request
2018-08-07 12:00:12 +02:00
Elad
93fe16b0a5 swarm/api/http: refactored http package (#17309) 2018-08-07 11:56:55 +02:00
Javier Peletier
64a4e89504 swarm/storage/mru: HOTFIX - fix panic in Handler.update (#17313) 2018-08-07 11:53:56 +02:00
Janoš Guljaš
a72ba5a55b cmd/swarm, swarm: various test fixes (#17299)
* swarm/network/simulation: increase the sleep duration for TestRun

* cmd/swarm, swarm: fix failing tests on mac

* cmd/swarm: update TestCLISwarmFs skip comment

* swarm/network/simulation: adjust disconnections on simulation close

* swarm/network/simulation: call cleanups after net shutdown
2018-08-06 12:33:22 +03:00
Péter Szilágyi
454382e81a
params, swarm/version: begin Geth v1.8.14, Swarm v0.3.2 cycle 2018-07-31 13:39:22 +03:00
Péter Szilágyi
225171a4bf
params, swarm/version: release Geth v1.8.13, Swarm 0.3.1 2018-07-31 13:35:53 +03:00
holisticode
d6efa69187 Merge netsim mig to master (#17241)
* swarm: merged stream-tests migration to develop

* swarm/network: expose simulation RandomUpNode to use in stream tests

* swarm/network: wait for subs in PeerEvents and fix stream.runSyncTest

* swarm: enforce waitkademlia for snapshot tests

* swarm: fixed syncer tests and snapshot_sync_test

* swarm: linting of simulation package

* swarm: address review comments

* swarm/network/stream: fix delivery_test bugs and refactor

* swarm/network/stream: addressed PR comments @janos

* swarm/network/stream: enforce waitKademlia, improve TestIntervals

* swarm/network/stream: TestIntervals not waiting for chunk to be stored
2018-07-30 22:55:25 +02:00
Anton Evangelatov
273c7a9dc4 swarm/api: remove ioutil.ReadAll for HandleGetFiles (#17276) 2018-07-30 12:19:26 +03:00
Anton Evangelatov
a5d5609e38 build: rename swarm deb package to ethereum-swarm; change swarm deb version from 1.8.x to 0.3.x (#16988)
* build: add support for different package and binary names

* build: bump up copyright date

* build: change default PackageName to empty string

* build, internal, swarm: enhance build/release process

* build: hack ethereum-swarm as a "depends" in deb package

* build/ci: remove redundant variables

* build, cmd, mobile, params, swarm: remove VERSION file; rename Version to VersionMeta;

* internal: remove VERSION() method which reads VERSION file

* build: fix VersionFilePath to Version

* Makefile: remove clean_go_build_cache.sh until it works

* Makefile: revert removal of clean_go_build_cache.sh
2018-07-30 11:56:40 +03:00
Viktor Trón
1064b9e691
Merge pull request #17233 from ethersphere/swarm-readme
swarm: README.md
2018-07-25 05:38:14 +02:00
Anton Evangelatov
14bdcdeab4 swarm/testutil: remove EnableMetrics 2018-07-23 18:45:39 +02:00
Javier Peletier
427316a707 swarm/storage/mru: Client-side MRU signatures (#784)
* swarm/storage/mru: Add embedded publickey and remove ENS dep

This commit breaks swarm, swarm/api...
but tests in swarm/storage/mru pass

* swarm: Refactor swarm, swarm/api to mru changes, make tests pass

* swarm/storage/mru: Remove self from recv, remove test ens vldtr

* swarm/storage/mru: Remove redundant test, expose ResourceHash mthd

* swarm/storage/mru: Make HeaderGetter mandatory + godoc fixes

* swarm/storage: Remove validator prefix for metadata chunk

* swarm/storage/mru: Use Address instead of PublicKey

* swarm/storage/mru: Change index from name to metadata chunk addr

* swarm/storage/mru: Refactor swarm/api/... to MRU index changes

* swarm/storage/mru: Refactor cleanup

* swarm/storage/mru: Rebase cleanup

* swarm: Use constructor for GenericSigner MRU in swarm.go

* swarm/storage: Change to BMTHash for MRU hashing

* swarm/storage: Reduce loglevel on chunk validator logs

* swarm/storage/mru: Delint

* swarm: MRU Rebase cleanup

* swarm/storage/mru: client-side mru signatures

Rebase to PR #668 and fix all conflicts

* swarm/storage/mru:  refactor and documentation

* swarm/resource/mru: error-checking  tests for parseUpdate/newUpdateChunk

* swarm/storage/mru: Added resourcemetadata tests

* swarm/storage/mru: Added tests  for UpdateRequest

* swarm/storage/mru: more test coverage for UpdateRequest and comments

* swarm/storage/mru: Avoid fake chunks in parseUpdate()

* swarm/storage/mru: Documented resource.go extensively

moved some functions where they make most sense

* swarm/storage/mru: increase test coverage for UpdateRequest and

variable name changes throughout to increase consistency

* swarm/storage/mru: moved default timestamp to NewCreateRequest-

* swarm/storage/mru: lookup refactor

* swarm/storage/mru: added comments and renamed raw flag to rawmru

* swarm/storage/mru: fix receiver typo

* swarm/storage/mru: refactored update chunk new/create

* swarm/storage/mru:  refactored signature digest to avoid malleability

* swarm/storage/mru: optimize update data serialization

* swarm/storage/mru: refactor and cleanup

* swarm/storage/mru: add timestamp struct and serialization

* swarm/storage/mru: fix lint error and mark some old code for deletion

* swarm/storage/mru: remove unnecessary variable

* swarm/storage/mru: Added more comments throughout

* swarm/storage/mru: Refactored metadata chunk layout + extensive error...

* swarm/storage/mru: refactor cli parser
Changed resource info output to JSON

* swarm/storage/mru: refactor serialization for extensibility

refactored error messages to NewErrorf

* swarm/storage/mru: Moved Signature to resource_sign.
Check Sign errors in server tests

* swarm/storage/mru: Remove isSafeName() checks

* swarm/storage/mru: scrubbed off all references to "block" for time

* swarm/storage/mru: removed superfluous isSynced() call.

* swarm/storage/mru: remove isMultihash() and ToSafeName functions

* swarm/storage/mru: various fixes and comments

* swarm/storage/mru: decoupled cli for independent create/update
* Made resource name optional
* Removed unused LookupPrevious

* swarm/storage/mru: Decoupled resource create / update & refactor

* swarm/storage/mru: Fixed some comments as per issues raised in PR #743

* swarm/storage/mru: Cosmetic changes as per #743 comments

* swarm/storage/mru: refct request encoder/decoder > marshal/unmarshal

* swarm/storage/mru: Cosmetic changes as per review in #748

* swarm/storage/mru: removed timestamp proof placeholder

* swarm/storage/mru: cosmetic/doc/fixes changes as per comments in #704

* swarm/storage/mru: removed unnecessary check in Handler.update

* swarm/storage/mru: Implemented Marshaler/Unmarshaler iface in Request

* swarm/storage/mru: Fixed linter error

* swarm/storage/mru: removed redundant address in signature digest

* swarm/storage/mru: fixed bug: LookupLatestVersionInPeriod not working

* swarm/storage/mru: Unfold Request creation API for create or update+create
set common time source for mru package

* swarm/api/http: fix HandleGetResource error variable shadowed
when requesting a resource that does not exist

* swarm/storage/mru: Add simple check to detect duplicate updates

* swarm/storage/mru: moved Multihash() to the right place.

* cmd/swarm: remove unneeded clientaccountmanager.go

* swarm/storage/mru: Changed some comments as per reviews in #784

* swarm/storage/mru: Made SignedResourceUpdate.GetDigest() public

* swarm/storage/mru: cosmetic changes as per comments in #784

* cmd/swarm: Inverted --multihash flag default

* swarm/storage/mru: removed Verify from SignedResourceUpdate.fromChunk

* swarm/storage/mru: Moved validation code out of serializer
Cosmetic / comment changes

* swarm/storage/mru: Added unit tests for UpdateLookup

* swarm/storage/mru: Increased coverage of metadata serialization

* swarm/storage/mru: Increased test coverage of updateHeader serializers

* swarm/storage/mru: Add resourceUpdate serializer test
2018-07-23 15:33:33 +02:00
Elad
0647c4de7b swarm/api/http: http package refactoring 1/5 and 2/5 2018-07-23 15:33:32 +02:00
Janoš Guljaš
dcaaa3c804 swarm: network simulation for swarm tests (#769)
* cmd/swarm: minor cli flag text adjustments

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* changed colour of landing page

* landing page reacts to enter keypress

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* swarm: propagate ctx to internal apis (#754)

* swarm/simnet: add basic node/service functions

* swarm/netsim: add buckets for global state and kademlia health check

* swarm/netsim: Use sync.Map as bucket and provide cleanup function for...

* swarm, swarm/netsim: adjust SwarmNetworkTest

* swarm/netsim: fix tests

* swarm: added visualization option to sim net redesign

* swarm/netsim: support multiple services per node

* swarm/netsim: remove redundant return statement

* swarm/netsim: add comments

* swarm: shutdown HTTP in Simulation.Close

* swarm: sim HTTP server timeout

* swarm/netsim: add more simulation methods and peer events examples

* swarm/netsim: add WaitKademlia example

* swarm/netsim: fix comments

* swarm/netsim: terminate peer events goroutines on simulation done

* swarm, swarm/netsim: naming updates

* swarm/netsim: return not healthy kademlias on WaitTillHealthy

* swarm: fix WaitTillHealthy call in testSwarmNetwork

* swarm/netsim: allow bucket to have any type for a key

* swarm: Added snapshots to new netsim

* swarm/netsim: add more tests for bucket

* swarm/netsim: move http related things into separate files

* swarm/netsim: add AddNodeWithService option

* swarm/netsim: add more tests and Start* methods

* swarm/netsim: add peer events and kademlia tests

* swarm/netsim: fix some tests flakiness

* swarm/netsim: improve random nodes selection, fix TestStartStop* tests

* swarm/netsim: remove time measurement from TestClose to avoid flakiness

* swarm/netsim: builder pattern for netsim HTTP server (#773)

* swarm/netsim: add connect related tests

* swarm/netsim: add comment for TestPeerEvents

* swarm: rename netsim package to network/simulation
2018-07-23 15:33:25 +02:00
lash
f5b128a5b3 swarm/fuse: Hotfix missing parantheses in statement 2018-07-23 15:31:02 +02:00
Viktor Trón
fd982d3f3b swarm/bmt: async section writer interface to BMT (#778)
- AsyncHasher implements AsyncWriter interface
 - add extra level for zerohashes in pool to lookup empty data hash
 - remove unused segment, hash and depth fields from Tree
 - Hash pkg function -> syncHash moved to test
 - add asyncHash helper func to tests using shuffle
 - add TestAsyncCorrectness to tests
 - add BenchmarkBMTAsync to tests
 - refactor benchmarks using subbenchmarks
 - improved comments
 - preinitialise base hashers on the nodes
2018-07-23 15:09:25 +02:00
Anton Evangelatov
4ea2d707f9 gitter: change ethereum/swarm to ethersphere/orange-lounge 2018-07-21 00:47:46 +02:00
Anton Evangelatov
ee8877509a swarm/readme: add link to code review guidelines 2018-07-19 13:27:31 +02:00
Anton Evangelatov
763e64cad8 swarm: readme.md 2018-07-19 12:13:03 +02:00
Anton Evangelatov
d483da766f swarm/network: bump up protocol versions due to wrapped message intro (#17170) 2018-07-13 17:40:55 +02:00
Anton Evangelatov
7c9314f231 swarm: integrate OpenTracing; propagate ctx to internal APIs (#17169)
* swarm: propagate ctx, enable opentracing

* swarm/tracing: log error when tracing is misconfigured
2018-07-13 17:40:28 +02:00
Anton Evangelatov
f7d3678c28 swarm/api/http: http package refactoring 1/5 and 2/5 (#17164) 2018-07-12 15:42:45 +02:00
Anton Evangelatov
b3711af051 swarm: ctx propagation; bmt fixes; pss generic notification framework (#17150)
* cmd/swarm: minor cli flag text adjustments

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: update description of swarm cmd

* swarm: added network ID test

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* swarm: propagate ctx to internal apis (#754)

* swarm/metrics: collect disk measurements

* swarm/bmt: fix io.Writer interface

  * Write now tolerates arbitrary variable buffers
  * added variable buffer tests
  * Write loop and finalise optimisation
  * refactor / rename
  * add tests for empty input

* swarm/pss: (UPDATE) Generic notifications package (#744)

swarm/pss: Generic package for creating pss notification svcs

* swarm: Adding context to more functions

* swarm/api: change colour of landing page in templates

* swarm/api: change landing page to react to enter keypress
2018-07-09 14:11:49 +02:00
lash
1f0f6f0272 swarm/pss: Hide big network tests under longrunning flag (#17074) 2018-06-25 16:11:47 +02:00
Balint Gabor
0a22ae5572
swarm/fuse: Disable fuse tests, they are flaky (#17072) 2018-06-25 14:04:01 +02:00
ethersphere
e187711c65 swarm: network rewrite merge 2018-06-21 21:10:31 +02:00
Felix Lange
0255951587 crypto: replace ToECDSAPub with error-checking func UnmarshalPubkey (#16932)
ToECDSAPub was unsafe because it returned a non-nil key with nil X, Y in
case of invalid input. This change replaces ToECDSAPub with
UnmarshalPubkey across the codebase.
2018-06-12 15:26:08 +02:00
Wuxiang
8f8774cf6d all: fix various typos (#16533)
* fix typo

* fix typo

* fix typo
2018-04-19 16:32:02 +03:00
thomasmodeneis
ba1030b6b8 build: enable goimports and varcheck linters (#16446) 2018-04-18 00:53:50 +02:00
Felix Lange
0fac705ed0
compression/rle: delete RLE compression (#16468) 2018-04-09 13:47:06 +02:00
Anton Evangelatov
746392cfd2 swarm/storage: disable treechunker test while it is flaky (#16254) 2018-03-05 17:23:33 +01:00
Elad Nachmias
b574b57766 swarm: give correct error on 0x hash prefix (#16195)
- added a case error struct that contains information about certain error cases
in which we would like to output more information to the client
- added a validation method that iterates and adds the information that is
stored in the error cases
2018-02-27 15:32:38 +02:00
cooganb
22fc6928d7 swarm: creates Swarm landing page for browser 'localhost:xxxx/' GET request when running Swarm (#15926)
* swarm: began work on GetHandleFile method re: issue #155

* swarm: now able to serve landing page template

* swarm: added landing page template

* swarm: landing page has working input

* swarm: fixed CSS issue in template

* swarm: deleted extra lines

* swarm: deleted time header and made redirect a relative path

* swarm: removed code mistakenly left
2018-02-26 09:56:40 +01:00
Anton Evangelatov
114738982e swarm/metrics: introduce metrics export flag (#16177) 2018-02-23 16:22:16 +01:00
Anton Evangelatov
dcca613a0b swarm: initial instrumentation (#15969)
* swarm: initial instrumentation with go-metrics

* swarm: initialise metrics collection and add ResettingTimer to HTTP requests

* swarm: update metrics flags names. remove redundant Timer.

* swarm: rename method for periodically updating gauges

* swarm: finalise metrics after feedback

* swarm/network: always init kad metrics containers

* swarm/network: off-by-one index in metrics containers

* swarm, metrics: resolved conflicts
2018-02-23 14:19:59 +01:00
Lewis Marshall
b677a07d36 swarm/api/http: Fix using deprecated bzzr scheme (#16152)
Without this, deprecated bzzr requests just return an empty response.

Signed-off-by: Lewis Marshall <lewis@lmars.net>
2018-02-23 14:09:01 +01:00
Janos Guljas
6a9730edaa swarm, cmd/swarm: Merge branch 'master' into multiple-ens-endpoints 2018-02-22 18:51:34 +01:00
Janos Guljas
a3a07350dc swarm, cmd/swarm: Merge branch 'master' into multiple-ens-endpoints 2018-02-22 14:23:17 +01:00
cooganb
4e61ed02e2 swarm: add favicon for Swarm templates served by browser (#15958)
* swarm: added script to HTML header to create favicon addresses #153

* swarm: moved data blob direclty into link tag, removed script

* swarm: added favicon info to other html templates

* swarm: fixing test errors

* swarm: fixing favicon test

* swarm: fixing travis tests

* swarm: added script to HTML header to create favicon addresses #153

* swarm: moved data blob direclty into link tag, removed script

* swarm: added favicon info to other html templates

* swarm: fixing test errors

* swarm: fixing favicon test

* swarm: fixing travis tests
2018-02-15 16:24:20 +02:00
Felix Lange
a5c0bbb4f4
all: update license information (#16089) 2018-02-14 13:49:11 +01:00
Furkan KAMACI
b8caba9709 various: remove redundant parentheses (#15793) 2018-01-03 14:14:47 +02:00
Janoš Guljaš
542d51895f swarm/api: url scheme bzz-hash to get hashes of swarm content (#15238) (#15715)
* swarm/api: url scheme bzz-hash to get hashes of swarm content (#15238)

Update URI to support bzz-hash scheme and handle such HTTP requests by
responding with hash of the content as a text/plain response.

* swarm/api: return hash of the content for bzz-hash:// requests

* swarm/api: revert "return hash of the content for bzz-hash:// requests"

Return hashes of the content that would be returned by bzz-raw
request.

* swarm/api/http: handle error in TestBzzGetPath

* swarm/api: remove extra blank line in comment
2017-12-21 14:47:10 +02:00
Janoš Guljaš
c786f75389 swarm: bzz-list, bzz-raw and bzz-immutable schemes (#15667)
* swarm/api: url scheme bzz-list for getting list of files from manifest

Replace query parameter list=true for listing all files contained
in a swarm manifest with a new URL scheme bzz-list.

* swarm: replaace bzzr and bzzi schemes with bzz-raw and bzz-immutable

New URI Shemes are added and old ones are deprecated, but not removed.
Old Schemes bzzr and bzzi are functional for backward compatibility.

* swarm/api: completely remove bzzr and bzzi schemes

Remove old schemes in favour of bzz-raw and
bzz-immutable.

* swarm/api: revert "completely remove bzzr and bzzi schemes"

Keep bzzr and bzzi schemes for backward compatibility. At least
until 0.3 swarm release.
2017-12-19 10:49:30 +02:00
Janos Guljas
0d6a735a72 swarm/api: implement NoResolverError with information about TLD
MultiResolver needs to provide information about TLD that has
no resolver configured for.
2017-12-18 23:07:48 +01:00
Janos Guljas
c0a4d9e1e6 cmd/swarm, swarm: disable ENS API by default
Specifying ENS API CLI flag, env variable or configuration
field is required for ENS resolving. Backward compatibility is
preserved with --ens-api="" CLI flag value.
2017-12-18 16:22:39 +01:00
Janos Guljas
47a8014559 cmd/swarm: Merge branch 'master' into multiple-ens-endpoints
Fix a conflict in cmd/swarm envVarsOverride function.
2017-12-14 10:36:12 +01:00
Janos Guljas
19982f9467 swarm, cmd/swarm: Merge branch 'master' into multiple-ens-endpoints
Merge with changes that implement config file PR #15548.

Field *EnsApi string* in swarm/api.Config is replaced with
*EnsAPIs []string*.

A new field *EnsDisabled bool* is added to swarm/api.Config for
easy way to disable ENS resolving with config file.

Signature of function swarm.NewSwarm is changed and simplified.
2017-12-13 10:40:39 +01:00
Zach
3da1bf8ca1 all: use gometalinter.v2, fix new gosimple issues (#15650) 2017-12-12 19:05:47 +01:00
holisticode
32516c768e cmd/swarm: add config file (#15548)
This commit adds a TOML configuration option to swarm. It reuses
the TOML configuration structure used in geth with swarm
customized items.

The commit:

* Adds a "dumpconfig" command to the swarm executable which
  allows printing the (default) configuration to stdout, which
  then can be redirected to a file in order to customize it.
* Adds a "--config <file>" option to the swarm executable which will
  allow to load a configuration file in TOML format from the
  specified location in order to initialize the Swarm node The
  override priorities are like follows: environment variables
  override command line arguments override config file override
  default config.
2017-12-11 22:56:06 +01:00
Janos Guljas
1dc19de5da swarm/api: use path.Ext instead strings.LastIndex in MultiResolver.Resolve 2017-12-04 22:56:52 +01:00
Janos Guljas
3732c15faa swarm/api: remove unneeded blank assignement 2017-12-04 22:32:33 +01:00
Janos Guljas
a758b5cf7a swarm/api: initialize map with make to comply with the convention 2017-12-04 22:31:00 +01:00
Janos Guljas
34edbc8868 swarm/api: remove unneeded assignment in MultiResolverOptionWithResolver 2017-12-04 22:29:37 +01:00
ferhat elmas
1d06e41f04 p2p, swarm/network/kademlia: use IsZero to check for zero time (#15603) 2017-12-04 11:07:10 +01:00
Janos Guljas
057af8c5c8 swarm: add CLI --ens-endpoint flag (#14386)
Implement a CLI flag that can be repeated to allow multiple ENS
resolvers for different TLDs.
2017-12-01 11:25:50 +01:00
Péter Szilágyi
f0ac925fa7
Merge pull request #15329 from holisticode/exact-match-fix
swarm/api: bug fix exact match for manifest
2017-11-24 16:23:37 +02:00
ferhat elmas
86f6568f66 build: enable unconvert linter (#15456)
* build: enable unconvert linter

 - fixes #15453
 - update code base for failing cases

* cmd/puppeth: replace syscall.Stdin with os.Stdin.Fd() for unconvert linter
2017-11-10 19:06:45 +02:00
Fabio Barone
03ec3fed2b swarm/api: bug fix exact match for manifest 2017-11-09 10:38:42 -05:00
ferhat elmas
9619a61024 all: gofmt -w -s (#15419) 2017-11-08 11:45:52 +01:00
ferhat elmas
07e8c177e7 core, swarm: typo fixes 2017-10-30 01:23:23 +01:00
Darrel Herbst
89860f4197 swarm/fuse: return amount of data written if the file exists (#15261)
If the file already existed, the WriteResponse.Size was being set
as the length of the entire file, not just the amount that was
written to the existing file.

Fixes #15216
2017-10-09 12:45:30 +02:00
holisticode
1ae0411d41 swarm/api: fixed 404 handling on missing default entry (#15139) 2017-10-06 15:45:54 +02:00
Zahoor Mohamed
d558a595ad swarm/storage: pyramid chunker re-write (#14382) 2017-09-21 22:22:51 +02:00
holisticode
ac193e36ce swarm/api/http: add error pages (#14967) 2017-09-08 20:29:09 +02:00
Fiisio
2dcb22afec swarm/fuse: use Equal instead of Compare (#15097) 2017-09-07 23:10:51 +02:00
Viktor Trón
2bacf36d80 bmt: Binary Merkle Tree Hash (#14334)
bmt is a new package that provides hashers for binary merkle tree hashes on
size-limited chunks. the main motivation is that using BMT hash as the chunk
hash of the swarm hash offers logsize inclusion proofs for arbitrary files on a
32-byte resolution completely viable to use in challenges on the blockchain.
2017-09-05 12:38:36 +02:00
Egon Elbre
133de3d806 swarm: fix megacheck warnings 2017-08-14 18:12:37 +03:00
Lewis Marshall
60c858a529 swarm/api: make api.NewManifest synchronous (#14880)
Previously, NewManifest was asynchronous so subsequent code which tried
to use the returned manifest could error as the manifest was not yet
persisted.
2017-07-31 15:58:19 +02:00
Lewis Marshall
e9b850805e cmd/swarm: support exporting, importing chunk db (#14868) 2017-07-31 13:23:44 +02:00
Lewis Marshall
f4841ff43d swarm/api/http: redirect root manifest requests to include trailing slash (#14806) 2017-07-25 11:51:26 +02:00
Lewis Marshall
c7041fe145 cmd/swarm: Exit if --ethapi is set
The previous attempt to use --ethapi as a fallback if --ens-api is not
set does not work because --ens-api has a default value, and also
setting --ens-api to "" is the suggested way to disable ENS lookups.

Signed-off-by: Lewis Marshall <lewis@lmars.net>
2017-06-30 10:16:03 +01:00
Lewis Marshall
f3359d5e58 cmd/swarm: Support using Mainnet for resolving ENS names
Signed-off-by: Lewis Marshall <lewis@lmars.net>
2017-06-26 12:51:33 +02:00
aron
caa00b7e6d swarm/storage: remove panic on invalid chunk 2017-06-25 16:10:54 +02:00
Felix Lange
98e101ef8e swarm/fuse: use subtests 2017-06-21 09:56:58 +02:00
Felix Lange
50c18e6eb8 swarm/fuse: simplify externalUnmount
The code looked for /usr/bin/diskutil on darwin, but it's actually
located in /usr/sbin. Fix that by not specifying the absolute path.
Also remove weird timeout construction and extra whitespace.
2017-06-21 09:56:58 +02:00
Lewis Marshall
0036e2a747 swarm/dev: add development environment (#14332)
This PR adds a Swarm development environment which can be run in a
Docker container and provides scripts for building binaries and running
Swarm clusters.
2017-06-01 12:52:18 +02:00
Péter Szilágyi
aa73420207
accounts/keystore, crypto: enforce 256 bit keys on import 2017-05-23 14:58:03 +03:00
Lewis Marshall
2a41e76b39 swarm/api: Fix adding paths which exist as manifests (#14482)
Signed-off-by: Lewis Marshall <lewis@lmars.net>
2017-05-22 08:57:03 +02:00
Lewis Marshall
4a2c17b1ab cmd/swarm: Add --httpaddr flag (#14475)
Fixes #14474.

Signed-off-by: Lewis Marshall <lewis@lmars.net>
2017-05-22 08:56:40 +02:00
Lewis Marshall
b0d0fafd68 swarm/api: fix error reporting in api.Resolve (#14464)
Previously, resolve errors were being swallowed and the returned error
was a generic "not a content hash" which isn't helpful.

This updates the Resolve function to fail fast rather than only
returning an error at the end, and also adds test coverage.
2017-05-13 02:02:25 +02:00
Felix Lange
0cc492f815 all: update license information 2017-04-14 10:29:00 +02:00
holisticode
409b61fe3c swarm/api: better name resolver handling (#3754)
Fixes #3608
2017-04-13 11:06:19 +02:00
Zahoor Mohamed
1d1d988aa7 swarm/api: FUSE read-write support (#13872)
- Moved fuse related code in a new package, swarm/fuse
- Added write support
  - Create new files
  - Delete existing files
  - Append to files (with limitations)
- More test coverage
2017-04-12 02:06:02 +02:00
Lewis Marshall
71fdaa4238 swarm/api: refactor and improve HTTP API (#3773)
This PR deprecates the file related RPC calls in favour of an improved HTTP API.

The main aim is to expose a simple to use API which can be consumed by thin
clients (e.g. curl and HTML forms) without the need for complex logic (e.g.
manipulating prefix trie manifests).
2017-04-07 00:22:22 +02:00
Louis Holbrook
0ec1104ba9 cmd/swarm: allow uploading from stdin (#3744)
- intended to be a swarm alternative to termbin.com
- added --stdin flag to swarm executable. if set, swarm will
  read data from stdin and postRaw it.
2017-04-06 14:21:16 +02:00
Lewis Marshall
b319f027a0 cmd/swarm, swarm/api/client: add HTTP API client and 'swarm ls' command (#3742)
This adds a swarm ls command which lists files and directories stored in a
manifest. Rather than listing all files, it uses "directory prefixes" in case there are a
lot of files in a manifest but you just want to traverse it.

This also includes some refactoring to the tests and the introduction of a
swarm/api/client package to make things easier to test.
2017-04-05 00:20:07 +02:00
Felix Lange
105b37f1b4 swarm/api: improve FUSE build constraints, logging and APIs (#3818)
* swarm/api: fix build/tests on unsupported platforms

Skip FUSE tests if FUSE is unavailable and change build constraints so
the 'lesser' platforms aren't mentioned explicitly. The test are
compiled on all platforms to prevent regressions in _fallback.go

Also gofmt -w -s because why not.

* internal/web3ext: fix swarmfs wrappers

Remove inputFormatter specifications so users get an error
when passing the wrong number of arguments.

* swarm/api: improve FUSE-related logging and APIs

The API now returns JSON objects instead of strings.
Log messages for invalid arguments are removed.
2017-03-31 12:11:01 +02:00
Zahoor Mohamed
11e7a712f4 swarm/api: support mounting manifests via FUSE (#3690) 2017-03-23 14:56:06 +01:00
Felix Lange
c213fd1fd8 all: import "context" instead of "golang.org/x/net/context"
There is no need to depend on the old context package now that the
minimum Go version is 1.7. The move to "context" eliminates our weird
vendoring setup. Some vendored code still uses golang.org/x/net/context
and it is now vendored in the normal way.

This change triggered new vet checks around context.WithTimeout which
didn't fire with golang.org/x/net/context.
2017-03-22 20:49:15 +01:00
holisticode
d97dea51ec swarm/network/kademlia: set kademlia log output to debug instead of warn (#3787) 2017-03-18 01:45:00 +01:00
Micah Zoltu
4e4e5fca54 rpc: add support for extended headers in CORS requests (#3783)
Fixes #3762.  Details about parameter:
a62a804a8a/cors.go (L50-L54)
2017-03-16 04:34:08 +01:00
Péter Szilágyi
7e3762fdc6
errs: kill it with fire 2017-02-28 18:01:54 +02:00
Felix Lange
5f7826270c all: unify big.Int zero checks, use common/math in more places (#3716)
* common/math: optimize PaddedBigBytes, use it more

name              old time/op    new time/op    delta
PaddedBigBytes-8    71.1ns ± 5%    46.1ns ± 1%  -35.15%  (p=0.000 n=20+19)

name              old alloc/op   new alloc/op   delta
PaddedBigBytes-8     48.0B ± 0%     32.0B ± 0%  -33.33%  (p=0.000 n=20+20)

* all: unify big.Int zero checks

Various checks were in use. This commit replaces them all with Int.Sign,
which is cheaper and less code.

eg templates:

    func before(x *big.Int) bool { return x.BitLen() == 0 }
    func after(x *big.Int) bool  { return x.Sign() == 0 }

    func before(x *big.Int) bool { return x.BitLen() > 0 }
    func after(x *big.Int) bool  { return x.Sign() != 0 }

    func before(x *big.Int) int { return x.Cmp(common.Big0) }
    func after(x *big.Int) int  { return x.Sign() }

* common/math, crypto/secp256k1: make ReadBits public in package math
2017-02-28 15:09:11 +01:00
Felix Lange
d0eba23af3 all: disable log message colors outside of geth
Also tweak behaviour so colors are only enabled when stderr is a terminal.
2017-02-27 15:33:12 +01:00
Péter Szilágyi
61e6bb1247
eth, les, swarm: fix go vet issues sufraced by log15 2017-02-23 12:16:45 +02:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Péter Szilágyi
3f923f3902
swarm: remove superfluous line break in log statements 2017-02-23 12:00:04 +02:00
Péter Szilágyi
e94dfb78f8
swarm/api/http: fix go vet issue on Go 1.8 2017-02-17 11:48:26 +02:00
Viktor Trón
e23e86921b swarm/network: fix chunk integrity checks (#3665)
* swarm/network: integrity on incoming known chunks
* swarm/network: fix integrity check for incoming chunks
* swarm/storage: imrpoved integrity checking on chunks
* dbstore panics on corrupt chunk entry an prompts user to run cleandb
* memstore adds logging for garbage collection
* dbstore refactor item delete. correct partial deletes in Get
* cmd/swarm: added cleandb subcommand
2017-02-13 13:20:50 +01:00
Maksim
6dd27e7cff swarm/storage: release chunk storage after stop swarm (#3651)
closes #3650
2017-02-08 18:01:12 +01:00
Péter Szilágyi
e5a93bf99a swarm/api/http: add missing copyright header 2017-01-30 15:21:46 +02:00
nolash
7669c5b5ec cmd/swarm, swarm/api: bzzr improve + networkid prio
fixes #3444
fixes #3494
networkid override

Added comments to explain why test against 0 appears twice

* Command line overrides saved config, saved config overrides system default

---

fixes #3476
bzzr get with path

Finally a hopefully clean commit for this PR
Added check for empty path to avoid SIGSEGV in path parser and resolver
Added requested tests for empty path and non-existing manifest.
However signature for StartHTTPServer had changed.
Now it's hacked as so:

	StartHttpServer(api.API, &Server{Addr: "127.0.0.1:8504", CorsString: ""})

* Parse url before resolve when path and ENS is supplied, example
* swarm/api/http proxy server test for retrieval of subpath through get
* Removed nil entry assignment on subtrie leaf in recursive key retrieval
* Cleaned up path-or-no-path condition in proxy server get handler
* swarm: processed with gofmt refers to lash/go-ethereum@90daa7a
* swarm: Added public access method Parse alias to parse
* swarm: processed with gofmt References nolash/go-ethereum@2ec3fd7
* Rename parse to Parse, removed alias
2017-01-27 08:18:13 +01:00
Bas van Kervel
54a65e6d87
cmd,eth,les,internal: remove natspec support 2017-01-17 12:13:50 +01:00
Felix Lange
b9b3efb09f all: fix ineffectual assignments and remove uses of crypto.Sha3
go get github.com/gordonklaus/ineffassign
ineffassign .
2017-01-09 16:24:42 +01:00
Péter Szilágyi
18c77744ff
all: fix spelling errors 2017-01-06 19:44:35 +02:00
Felix Lange
13e3b2f433 logger, pow/dagger, pow/ezp: delete dead code 2017-01-06 18:18:07 +01:00
Felix Lange
f2da6581ba all: fix issues reported by honnef.co/go/simple/cmd/gosimple 2017-01-06 18:18:07 +01:00
Felix Lange
35a7dcb162 all: gofmt -w -s 2017-01-06 15:52:03 +01:00
Maran
f087633efd swarm/api/http: add support for CORS headers (#3388) 2017-01-05 11:57:41 +01:00
Aron Fischer
9a51f5c350 swarm/http: check error returned by reader.Size (#3470) 2016-12-22 00:35:05 +01:00
Aron Fischer
301c0a6303 swarm/storage: call size before seek-from-end (#3469) 2016-12-22 00:34:05 +01:00
Aron Fischer
65f486ff02 swarm/api: check for zero length manifest error (#3468) 2016-12-22 00:32:08 +01:00
zelig
4181046488 swarm/network, cmd/swarm: swarm default network id is 3 (to match Ropsten) 2016-12-13 12:54:13 +01:00
Nick Johnson
671fd94e25 swarm/api: Update ENS root address for Ropsten & prod (#3391) 2016-12-01 19:33:10 +01:00
Viktor Trón
e949a2ed2f cmd/bzzd: swarm daemon fixes (#3359)
* cmd/bzzd: add missing p2p/discovery flags
* cmd/bzzd: fix two bugs crashing bzzd if bootnodes flag given
* cmd/bzzd: make no swap default, renamed flag bzznoswap->bzzswap
* internal/web3ext: correct methods for bzz IPC module
* cmd/bzzd: ethapi param not mandatory. Warning if no blockchain
* cmd/bzzd: correct default IPC modules in help string
* cmd/utils: fix help description for networkId - add Ropsten
* cmd/bzz, swarm/api, swarm/network: add swarm networkId flag
* cmd/bzzd: change nosync flag to sync and BootTFlag
2016-11-28 13:29:33 +01:00
Felix Lange
d1a95c643e Merge pull request #3325 from fjl/p2p-netrestrict
Prevent relay of invalid IPs, add --netrestrict
2016-11-25 13:59:18 +01:00
Jeffrey Wilcke
67e0894d9e common/httpclient, les: removed httpclient 2016-11-25 13:10:44 +01:00
Felix Lange
e46bda5093 swarm/network: prevent relay of invalid IPs 2016-11-22 22:21:18 +01:00
Felix Lange
5cd4430a8d swarm/api/http: reject requests without content-length 2016-11-11 00:15:21 +01:00
Felix Lange
23420ff67a swarm/network: skip TestSyncDb 2016-11-11 00:15:21 +01:00
Felix Lange
ef50e01c5e swarm/api: add bzz prefix to swarm directory 2016-11-11 00:15:20 +01:00
Felix Lange
b8bd9a71c8 all: update license information 2016-11-09 02:51:34 +01:00
Kenji Siu
d85d3c74db internal/debug, internal/ethapi, rpc, swarm/storage: Ran "go fmt" 2016-11-01 18:40:36 +08:00
Péter Szilágyi
81b01f1c2b Merge pull request #3111 from obscuren/gas-price-fork
core, core/vm: added gas price variance table (EIP #150)
2016-10-14 19:32:11 +03:00
Jeffrey Wilcke
64af2aafda core, core/vm: added gas price variance table
This implements 1b & 1c of EIP150 by adding a new GasTable which must be
returned from the RuleSet config method. This table is used to determine
the gas prices for the current epoch.

Please note that when the CreateBySuicide gas price is set it is assumed
that we're in the new epoch phase.

In addition this PR will serve as temporary basis while refactorisation
in being done in the EVM64 PR, which will substentially overhaul the gas
price code.
2016-10-14 18:09:17 +02:00
Daniel A. Nagy
eb2f01aee8 swarm/storage: Allow EOF at the end of the reader in the chunker. Handle the case when Read returns less than length of target slice 2016-10-13 21:23:28 +02:00
aron
a45421baaf swarm/storage: fix chunker when reader is broken
* brokenLimitedReader gives error after half size
* TestRandomBrokenData tests chunker with broken reader
* add blocking quitC (instead of errC) and use errC only for errors
* don't close chunkC in tester Split,
* use quitC to quit chunk storage loop
2016-10-10 23:34:44 +02:00
ΞTHΞЯSPHΞЯΞ
4d300e4dec swarm: plan bee for content storage and distribution on web3
This change imports the Swarm protocol codebase. Compared to the 'swarm'
branch, a few mostly cosmetic changes had to be made:

* The various redundant log message prefixes are gone.
* All files now have LGPLv3 license headers.
* Minor code changes were needed to please go vet and make the tests
  pass on Windows.
* Further changes were required to adapt to the go-ethereum develop
  branch and its new Go APIs.

Some code has not (yet) been brought over:

* swarm/cmd/bzzhash: will reappear as cmd/bzzhash later
* swarm/cmd/bzzup.sh: will be reimplemented in cmd/bzzup
* swarm/cmd/makegenesis: will reappear somehow
* swarm/examples/album: will move to a separate repository
* swarm/examples/filemanager: ditto
* swarm/examples/files: will not be merged
* swarm/test/*: will not be merged
* swarm/services/swear: will reappear as contracts/swear when needed
2016-08-31 16:19:40 +02:00