cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity

Merge branch 'master' into nonsense/cli-show-deals

Browse Source
This commit is contained in:
Anton Evangelatov 2021-03-23 13:58:26 +02:00
commit 22217b7cae
150 changed files with 6518 additions and 1452 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.circleci/config.yml
View File

@ -408,9 +408,18 @@ jobs:
steps:
- install-deps
- prepare
- run: zcat build/openrpc/full.json.gz | jq > ../pre-openrpc-full
- run: zcat build/openrpc/miner.json.gz | jq > ../pre-openrpc-miner
- run: zcat build/openrpc/worker.json.gz | jq > ../pre-openrpc-worker
- run: make deps
- run: make docsgen
- run: zcat build/openrpc/full.json.gz | jq > ../post-openrpc-full
- run: zcat build/openrpc/miner.json.gz | jq > ../post-openrpc-miner
- run: zcat build/openrpc/worker.json.gz | jq > ../post-openrpc-worker
- run: git --no-pager diff
- run: diff ../pre-openrpc-full ../post-openrpc-full
- run: diff ../pre-openrpc-miner ../post-openrpc-miner
- run: diff ../pre-openrpc-worker ../post-openrpc-worker
- run: git --no-pager diff --quiet
lint: &lint

33
.github/ISSUE_TEMPLATE/bug-report.md vendored Normal file
View File

@ -0,0 +1,33 @@
---
name: Bug Report
about: Create a report to help us improve
title: "[BUG] "
labels: hint/needs-triaging, kind/bug
assignees: ''
---
> Note: For security-related bugs/issues, please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
**Describe the bug**
A clear and concise description of what the bug is.
(If you are not sure what the bug is, try to figure it out via a [discussion](https://github.com/filecoin-project/lotus/discussions/new) first!
**Version (run `lotus version`):**
**To Reproduce**
Steps to reproduce the behavior:
1. Run '...'
2. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Logs**
Provide daemon/miner/worker logs, and goroutines(if available) for troubleshooting.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Add any other context about the problem here.

27
.github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@ -1,27 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Run '...'
2. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Version (run `lotus version`):**
**Additional context**
Add any other context about the problem here.

49
.github/ISSUE_TEMPLATE/deal-making-issues.md vendored Normal file
View File

@ -0,0 +1,49 @@
---
name: Deal Making Issues
about: Create a report for help with deal making failures.
title: "[Deal Making Issue]"
labels: hint/needs-triaging, area/markets
assignees: ''
---
> Note: For security-related bugs/issues, please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
Please provide all the information requested here to help us troubleshoot "deal making failed" issues.
If the information requested is missing, we will probably have to just ask you to provide it anyway,
before we can help debug.
**Basic Information**
Including information like, Are you the client or the miner? Is this a storage deal or a retrieval deal? Is it an offline deal?
**Describe the problem**
A brief description of the problem you encountered while trying to make a deal.
**Version**
The output of `lotus --version`.
**Setup**
You miner(if applicable) and daemon setup, i.e: What hardware do you use, how much ram and etc.
**To Reproduce**
Steps to reproduce the behavior:
1. Run '...'
2. See error
**Deal status**
The output of `lotus client list-deals -v` and/or `lotus-miner storage-deals|retrieval-deals|data-transfers list [-v]` commands for the deal(s) in question.
**Lotus daemon and miner logs**
Please go through the logs of your daemon and miner(if applicable), and include screenshots of any error/warning-like messages you find.
Alternatively please upload full log files and share a link here
** Code modifications **
If you have modified parts of lotus, please describe which areas were modified,
and the scope of those modifications

20
.github/ISSUE_TEMPLATE/feature_request.md vendored Normal file
View File

@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: "[Feature Request]"
labels: hint/needs-triaging
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

35
.github/ISSUE_TEMPLATE/mining-issues.md vendored Normal file
View File

@ -0,0 +1,35 @@
---
name: Mining Issues
about: Create a report for help with mining failures.
title: "[Mining Issue]"
labels: hint/needs-triaging, area/mining
assignees: ''
---
> Note: For security-related bugs/issues, please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
Please provide all the information requested here to help us troubleshoot "mining/WinningPoSt failed" issues.
If the information requested is missing, you may be asked you to provide it.
**Describe the problem**
A brief description of the problem you encountered while mining new blocks.
**Version**
The output of `lotus --version`.
**Setup**
You miner and daemon setup, including what hardware do you use, your environment variable settings, how do you run your miner and worker, do you use GPU and etc.
**Lotus daemon and miner logs**
Please go through the logs of your daemon and miner, and include screenshots of any error/warning-like messages you find, highlighting the one has "winning post" in it.
Alternatively please upload full log files and share a link here
** Code modifications **
If you have modified parts of lotus, please describe which areas were modified,
and the scope of those modifications

46
.github/ISSUE_TEMPLATE/proving-issues.md vendored Normal file
View File

@ -0,0 +1,46 @@
---
name: Proving Issues
about: Create a report for help with proving failures.
title: "[Proving Issue]"
labels: area/proving, hint/needs-triaging
assignees: ''
---
> Note: For security-related bugs/issues, please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
Please provide all the information requested here to help us troubleshoot "proving/window PoSt failed" issues.
If the information requested is missing, we will probably have to just ask you to provide it anyway,
before we can help debug.
**Describe the problem**
A brief description of the problem you encountered while proving the storage.
**Version**
The output of `lotus --version`.
**Setup**
You miner and daemon setup, including what hardware do you use, your environment variable settings, how do you run your miner and worker, do you use GPU and etc.
**Proving status**
The output of `lotus-miner proving` info.
**Lotus miner logs**
Please go through the logs of your miner, and include screenshots of any error-like messages you find, highlighting the one has "window post" in it.
Alternatively please upload full log files and share a link here
**Lotus miner diagnostic info**
Please collect the following diagnostic information, and share a link here
* lotus-miner diagnostic info `lotus-miner info all > allinfo.txt`
** Code modifications **
If you have modified parts of lotus, please describe which areas were modified,
and the scope of those modifications

23
.github/ISSUE_TEMPLATE/sealingfailed.md → .github/ISSUE_TEMPLATE/sealing-issues.md vendored
View File

@ -1,21 +1,32 @@
---
name: Sealing Issues
about: Create a report for help with sealing (commit) failures.
title: ''
labels: 'sealing'
title: "[Sealing Issue]"
labels: hint/needs-triaging, area/sealing
assignees: ''
---
> Note: For security-related bugs/issues, please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
Please provide all the information requested here to help us troubleshoot "commit failed" issues.
If the information requested is missing, we will probably have to just ask you to provide it anyway,
before we can help debug.
**Describe the problem**
A brief description of the problem you encountered while sealing a sector.
A brief description of the problem you encountered while proving (sealing) a sector.
**Version**
Including what commands you ran, and a description of your setup, is very helpful.
The output of `lotus --version`.
**Setup**
You miner and daemon setup, including what hardware do you use, your environment variable settings, how do you run your miner and worker, do you use GPU and etc.
**Commands**
Commands you ran.
**Sectors status**
@ -37,7 +48,3 @@ Please collect the following diagnostic information, and share a link here
If you have modified parts of lotus, please describe which areas were modified,
and the scope of those modifications
**Version**
The output of `lotus --version`.

2
.gitignore vendored
View File

@ -14,6 +14,8 @@
/lotus-pcr
/lotus-wallet
/lotus-keygen
/docgen-md
/docgen-openrpc
/bench.json
/lotuspond/front/node_modules
/lotuspond/front/build

87
CHANGELOG.md
View File

@ -1,5 +1,92 @@
# Lotus changelog
# 1.5.2 / 2021-03-11
This is an hotfix release of Lotus that fixes a critical bug introduced in v1.5.1 in the miner windowPoSt logic. This upgrade is only affecting miner nodes.
## Changes
- fix window post rand check (https://github.com/filecoin-project/lotus/pull/5773)
- wdpost: Always use head tipset to get randomness (https://github.com/filecoin-project/lotus/pull/5774)
# 1.5.1 / 2021-03-10
This is an optional release of Lotus that introduces an important fix to the WindowPoSt computation process. The change is to wait for some confidence before drawing beacon randomness for the proof. Without this, invalid proofs might be generated as the result of a null tipset.
## Splitstore
This release also introduces the splitstore, a new optional blockstore that segregates the monolithic blockstore into cold and hot regions. The hot region contains objects from the last 4-5 finalities plus all reachable objects from two finalities away. All other objects are moved to the cold region using a compaction process that executes every finality, once 5 finalities have elapsed.
The splitstore allows us to separate the two regions quite effectively, using two separate badger blockstores. The separation
means that the live working set is much smaller, which results in potentially significant performance improvements. In addition, it means that the coldstore can be moved to a separate (bigger, slower, cheaper) disk without loss of performance.
The design also allows us to use different implementations for the two blockstores; for example, an append-only blockstore could be used for coldstore and a faster memory mapped blockstore could be used for the hotstore (eg LMDB). We plan to experiment with these options in the future.
Once the splitstore has been enabled, the existing monolithic blockstore becomes the coldstore. On the first head change notification, the splitstore will warm up the hotstore by copying all reachable objects from the current tipset into the hotstore. All new writes go into the hotstore, with the splitstore tracking the write epoch. Once 5 finalities have elapsed, and every finality thereafter, the splitstore compacts by moving cold objects into the coldstore. There is also experimental support for garbage collection, whereby nunreachable objects are simply discarded.
To enable the splitstore, add the following to config.toml:
```
[Chainstore]
EnableSplitstore = true
```
## Highlights
Other highlights include:
- Improved deal data handling - now multiple deals can be adding to sectors in parallel
- Rewriten sector pledging - it now actually cares about max sealing sector limits
- Better handling for sectors stuck in the RecoverDealIDs state
- lotus-miner sectors extend command
- Optional configurable storage path size limit
- Config to disable owner/worker fallback from control addresses (useful when owner is a key on a hardware wallet)
- A write log for node metadata, which can be restored as a backup when the metadata leveldb becomes corrupted (e.g. when you run out of disk space / system crashes in some bad way)
## Changes
- avoid use mp.cfg directly to avoid race (https://github.com/filecoin-project/lotus/pull/5350)
- Show replacing message CID is state search-msg cli (https://github.com/filecoin-project/lotus/pull/5656)
- Fix riceing by importing the main package (https://github.com/filecoin-project/lotus/pull/5675)
- Remove sectors with all deals expired in RecoverDealIDs (https://github.com/filecoin-project/lotus/pull/5658)
- storagefsm: Rewrite input handling (https://github.com/filecoin-project/lotus/pull/5375)
- reintroduce Refactor send command for better testability (https://github.com/filecoin-project/lotus/pull/5668)
- Improve error message with importing a chain (https://github.com/filecoin-project/lotus/pull/5669)
- storagefsm: Cleanup CC sector creation (https://github.com/filecoin-project/lotus/pull/5612)
- chain list --gas-stats display capacity (https://github.com/filecoin-project/lotus/pull/5676)
- Correct some logs (https://github.com/filecoin-project/lotus/pull/5694)
- refactor blockstores (https://github.com/filecoin-project/lotus/pull/5484)
- Add idle to sync stage's String() (https://github.com/filecoin-project/lotus/pull/5702)
- packer provisioner (https://github.com/filecoin-project/lotus/pull/5604)
- add DeleteMany to Blockstore interface (https://github.com/filecoin-project/lotus/pull/5703)
- segregate chain and state blockstores (https://github.com/filecoin-project/lotus/pull/5695)
- fix(multisig): The format of the amount is not correct in msigLockApp (https://github.com/filecoin-project/lotus/pull/5718)
- Update butterfly network (https://github.com/filecoin-project/lotus/pull/5627)
- Collect worker task metrics (https://github.com/filecoin-project/lotus/pull/5648)
- Correctly format disputer log (https://github.com/filecoin-project/lotus/pull/5716)
- Log block CID in the large delay warning (https://github.com/filecoin-project/lotus/pull/5704)
- Move api client builders to a cliutil package (https://github.com/filecoin-project/lotus/pull/5728)
- Implement net peers --extended (https://github.com/filecoin-project/lotus/pull/5734)
- Command to extend sector expiration (https://github.com/filecoin-project/lotus/pull/5666)
- garbage collect hotstore after compaction (https://github.com/filecoin-project/lotus/pull/5744)
- tune badger gc to repeatedly gc the value log until there is no rewrite (https://github.com/filecoin-project/lotus/pull/5745)
- Add configuration option for pubsub IPColocationWhitelist subnets (https://github.com/filecoin-project/lotus/pull/5735)
- hot/cold blockstore segregation (aka. splitstore) (https://github.com/filecoin-project/lotus/pull/4992)
- Customize verifreg root key and remainder account when making genesis (https://github.com/filecoin-project/lotus/pull/5730)
- chore: update go-graphsync to 0.6.0 (https://github.com/filecoin-project/lotus/pull/5746)
- Add connmgr metadata to NetPeerInfo (https://github.com/filecoin-project/lotus/pull/5749)
- test: attempt to make the splitstore test deterministic (https://github.com/filecoin-project/lotus/pull/5750)
- Feat/api no dep build (https://github.com/filecoin-project/lotus/pull/5729)
- Fix bootstrapper profile setting (https://github.com/filecoin-project/lotus/pull/5756)
- Check liveness of sectors when processing termination batches (https://github.com/filecoin-project/lotus/pull/5759)
- Configurable storage path storage limit (https://github.com/filecoin-project/lotus/pull/5624)
- miner: Config to disable owner/worker address fallback (https://github.com/filecoin-project/lotus/pull/5620)
- Fix TestUnpadReader on Go 1.16 (https://github.com/filecoin-project/lotus/pull/5761)
- Metadata datastore log (https://github.com/filecoin-project/lotus/pull/5755)
- Remove the SR2 stats, leave just the network totals (https://github.com/filecoin-project/lotus/pull/5757)
- fix: wait a bit before starting to compute window post proofs (https://github.com/filecoin-project/lotus/pull/5764)
- fix: retry proof when randomness changes (https://github.com/filecoin-project/lotus/pull/5768)
# 1.5.0 / 2021-02-23
This is a mandatory release of Lotus that introduces the fifth upgrade to the Filecoin network. The network upgrade occurs at height 550321, before which time all nodes must have updated to this release (or later). At this height, [v3 specs-actors](https://github.com/filecoin-project/specs-actors/releases/tag/v3.0.0) will take effect, which in turn implements the following two FIPs:

43
Makefile
View File

@ -63,20 +63,23 @@ CLEAN+=build/.update-modules
deps: $(BUILD_DEPS)
.PHONY: deps
build-devnets: build lotus-seed lotus-shed lotus-wallet lotus-gateway
.PHONY: build-devnets
debug: GOFLAGS+=-tags=debug
debug: lotus lotus-miner lotus-worker lotus-seed
debug: build-devnets
2k: GOFLAGS+=-tags=2k
2k: lotus lotus-miner lotus-worker lotus-seed
2k: build-devnets
calibnet: GOFLAGS+=-tags=calibnet
calibnet: lotus lotus-miner lotus-worker lotus-seed
calibnet: build-devnets
nerpanet: GOFLAGS+=-tags=nerpanet
nerpanet: lotus lotus-miner lotus-worker lotus-seed
nerpanet: build-devnets
butterflynet: GOFLAGS+=-tags=butterflynet
butterflynet: lotus lotus-miner lotus-worker lotus-seed
butterflynet: build-devnets
lotus: $(BUILD_DEPS)
rm -f lotus
@ -324,10 +327,32 @@ method-gen:
gen: type-gen method-gen
docsgen:
go run ./api/docgen "api/api_full.go" "FullNode" > documentation/en/api-methods.md
go run ./api/docgen "api/api_storage.go" "StorageMiner" > documentation/en/api-methods-miner.md
go run ./api/docgen "api/api_worker.go" "WorkerAPI" > documentation/en/api-methods-worker.md
docsgen: docsgen-md docsgen-openrpc
docsgen-md-bin:
go build $(GOFLAGS) -o docgen-md ./api/docgen/cmd
docsgen-openrpc-bin:
go build $(GOFLAGS) -o docgen-openrpc ./api/docgen-openrpc/cmd
docsgen-md: docsgen-md-full docsgen-md-storage docsgen-md-worker
docsgen-md-full: docsgen-md-bin
./docgen-md "api/api_full.go" "FullNode" > documentation/en/api-methods.md
docsgen-md-storage: docsgen-md-bin
./docgen-md "api/api_storage.go" "StorageMiner" > documentation/en/api-methods-miner.md
docsgen-md-worker: docsgen-md-bin
./docgen-md "api/api_worker.go" "WorkerAPI" > documentation/en/api-methods-worker.md
docsgen-openrpc: docsgen-openrpc-full docsgen-openrpc-storage docsgen-openrpc-worker
docsgen-openrpc-full: docsgen-openrpc-bin
./docgen-openrpc "api/api_full.go" "FullNode" -gzip > build/openrpc/full.json.gz
docsgen-openrpc-storage: docsgen-openrpc-bin
./docgen-openrpc "api/api_storage.go" "StorageMiner" -gzip > build/openrpc/miner.json.gz
docsgen-openrpc-worker: docsgen-openrpc-bin
./docgen-openrpc "api/api_worker.go" "WorkerAPI" -gzip > build/openrpc/worker.json.gz
.PHONY: docsgen docsgen-md-bin docsgen-openrpc-bin
print-%:
@echo $*=$($*)

16
api/api_common.go
View File

@ -12,7 +12,7 @@ import (
"github.com/libp2p/go-libp2p-core/peer"
protocol "github.com/libp2p/go-libp2p-core/protocol"
"github.com/filecoin-project/lotus/build"
apitypes "github.com/filecoin-project/lotus/api/types"
)
type Common interface {
@ -33,6 +33,7 @@ type Common interface {
NetPubsubScores(context.Context) ([]PubsubScore, error)
NetAutoNatStatus(context.Context) (NatInfo, error)
NetAgentVersion(ctx context.Context, p peer.ID) (string, error)
NetPeerInfo(context.Context, peer.ID) (*ExtendedPeerInfo, error)
// NetBandwidthStats returns statistics about the nodes total bandwidth
// usage and current rate across all peers and protocols.
@ -53,11 +54,14 @@ type Common interface {
// MethodGroup: Common
// Discover returns an OpenRPC document describing an RPC API.
Discover(ctx context.Context) (apitypes.OpenRPCDocument, error)
// ID returns peerID of libp2p node backing this API
ID(context.Context) (peer.ID, error)
// Version provides information about API provider
Version(context.Context) (Version, error)
Version(context.Context) (APIVersion, error)
LogList(context.Context) ([]string, error)
LogSetLevel(context.Context, string, string) error
@ -71,15 +75,15 @@ type Common interface {
Closing(context.Context) (<-chan struct{}, error)
}
// Version provides various build-time information
type Version struct {
// APIVersion provides various build-time information
type APIVersion struct {
Version string
// APIVersion is a binary encoded semver version of the remote implementing
// this api
//
// See APIVersion in build/version.go
APIVersion build.Version
APIVersion Version
// TODO: git commit / os / genesis cid?
@ -87,7 +91,7 @@ type Version struct {
BlockDelay uint64
}
func (v Version) String() string {
func (v APIVersion) String() string {
return fmt.Sprintf("%s+api%s", v.Version, v.APIVersion.String())
}

129
api/api_full.go
View File

@ -67,10 +67,22 @@ type FullNode interface {
ChainGetTipSet(context.Context, types.TipSetKey) (*types.TipSet, error)
// ChainGetBlockMessages returns messages stored in the specified block.
//
// Note: If there are multiple blocks in a tipset, it's likely that some
// messages will be duplicated. It's also possible for blocks in a tipset to have
// different messages from the same sender at the same nonce. When that happens,
// only the first message (in a block with lowest ticket) will be considered
// for execution
//
// NOTE: THIS METHOD SHOULD ONLY BE USED FOR GETTING MESSAGES IN A SPECIFIC BLOCK
//
// DO NOT USE THIS METHOD TO GET MESSAGES INCLUDED IN A TIPSET
// Use ChainGetParentMessages, which will perform correct message deduplication
ChainGetBlockMessages(ctx context.Context, blockCid cid.Cid) (*BlockMessages, error)
// ChainGetParentReceipts returns receipts for messages in parent tipset of
// the specified block.
// the specified block. The receipts in the list returned is one-to-one with the
// messages returned by a call to ChainGetParentMessages with the same blockCid.
ChainGetParentReceipts(ctx context.Context, blockCid cid.Cid) ([]*types.MessageReceipt, error)
// ChainGetParentMessages returns messages stored in parent tipset of the
@ -349,7 +361,22 @@ type FullNode interface {
// tipset.
StateCall(context.Context, *types.Message, types.TipSetKey) (*InvocResult, error)
// StateReplay replays a given message, assuming it was included in a block in the specified tipset.
// If no tipset key is provided, the appropriate tipset is looked up.
//
// If a tipset key is provided, and a replacing message is found on chain,
// the method will return an error saying that the message wasn't found
//
// If no tipset key is provided, the appropriate tipset is looked up, and if
// the message was gas-repriced, the on-chain message will be replayed - in
// that case the returned InvocResult.MsgCid will not match the Cid param
//
// If the caller wants to ensure that exactly the requested message was executed,
// they MUST check that InvocResult.MsgCid is equal to the provided Cid.
// Without this check both the requested and original message may appear as
// successfully executed on-chain, which may look like a double-spend.
//
// A replacing message is a message with a different CID, any of Gas values, and
// different signature, but with all other parameters matching (source/destination,
// nonce, params, etc.)
StateReplay(context.Context, types.TipSetKey, cid.Cid) (*InvocResult, error)
// StateGetActor returns the indicated actor's nonce and balance.
StateGetActor(ctx context.Context, actor address.Address, tsk types.TipSetKey) (*types.Actor, error)
@ -402,15 +429,71 @@ type FullNode interface {
// StateSectorPartition finds deadline/partition with the specified sector
StateSectorPartition(ctx context.Context, maddr address.Address, sectorNumber abi.SectorNumber, tok types.TipSetKey) (*miner.SectorLocation, error)
// StateSearchMsg searches for a message in the chain, and returns its receipt and the tipset where it was executed
//
// NOTE: If a replacing message is found on chain, this method will return
// a MsgLookup for the replacing message - the MsgLookup.Message will be a different
// CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
// result of the execution of the replacing message.
//
// If the caller wants to ensure that exactly the requested message was executed,
// they MUST check that MsgLookup.Message is equal to the provided 'cid'.
// Without this check both the requested and original message may appear as
// successfully executed on-chain, which may look like a double-spend.
//
// A replacing message is a message with a different CID, any of Gas values, and
// different signature, but with all other parameters matching (source/destination,
// nonce, params, etc.)
StateSearchMsg(context.Context, cid.Cid) (*MsgLookup, error)
// StateSearchMsgLimited looks back up to limit epochs in the chain for a message, and returns its receipt and the tipset where it was executed
//
// NOTE: If a replacing message is found on chain, this method will return
// a MsgLookup for the replacing message - the MsgLookup.Message will be a different
// CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
// result of the execution of the replacing message.
//
// If the caller wants to ensure that exactly the requested message was executed,
// they MUST check that MsgLookup.Message is equal to the provided 'cid'.
// Without this check both the requested and original message may appear as
// successfully executed on-chain, which may look like a double-spend.
//
// A replacing message is a message with a different CID, any of Gas values, and
// different signature, but with all other parameters matching (source/destination,
// nonce, params, etc.)
StateSearchMsgLimited(ctx context.Context, msg cid.Cid, limit abi.ChainEpoch) (*MsgLookup, error)
// StateWaitMsg looks back in the chain for a message. If not found, it blocks until the
// message arrives on chain, and gets to the indicated confidence depth.
//
// NOTE: If a replacing message is found on chain, this method will return
// a MsgLookup for the replacing message - the MsgLookup.Message will be a different
// CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
// result of the execution of the replacing message.
//
// If the caller wants to ensure that exactly the requested message was executed,
// they MUST check that MsgLookup.Message is equal to the provided 'cid'.
// Without this check both the requested and original message may appear as
// successfully executed on-chain, which may look like a double-spend.
//
// A replacing message is a message with a different CID, any of Gas values, and
// different signature, but with all other parameters matching (source/destination,
// nonce, params, etc.)
StateWaitMsg(ctx context.Context, cid cid.Cid, confidence uint64) (*MsgLookup, error)
// StateWaitMsgLimited looks back up to limit epochs in the chain for a message.
// If not found, it blocks until the message arrives on chain, and gets to the
// indicated confidence depth.
//
// NOTE: If a replacing message is found on chain, this method will return
// a MsgLookup for the replacing message - the MsgLookup.Message will be a different
// CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
// result of the execution of the replacing message.
//
// If the caller wants to ensure that exactly the requested message was executed,
// they MUST check that MsgLookup.Message is equal to the provided 'cid'.
// Without this check both the requested and original message may appear as
// successfully executed on-chain, which may look like a double-spend.
//
// A replacing message is a message with a different CID, any of Gas values, and
// different signature, but with all other parameters matching (source/destination,
// nonce, params, etc.)
StateWaitMsgLimited(ctx context.Context, cid cid.Cid, confidence uint64, limit abi.ChainEpoch) (*MsgLookup, error)
// StateListMiners returns the addresses of every miner that has claimed power in the Power Actor
StateListMiners(context.Context, types.TipSetKey) ([]address.Address, error)
@ -431,13 +514,51 @@ type FullNode interface {
// StateChangedActors returns all the actors whose states change between the two given state CIDs
// TODO: Should this take tipset keys instead?
StateChangedActors(context.Context, cid.Cid, cid.Cid) (map[string]types.Actor, error)
// StateGetReceipt returns the message receipt for the given message
// StateGetReceipt returns the message receipt for the given message or for a
// matching gas-repriced replacing message
//
// NOTE: If the requested message was replaced, this method will return the receipt
// for the replacing message - if the caller needs the receipt for exactly the
// requested message, use StateSearchMsg().Receipt, and check that MsgLookup.Message
// is matching the requested CID
//
// DEPRECATED: Use StateSearchMsg, this method won't be supported in v1 API
StateGetReceipt(context.Context, cid.Cid, types.TipSetKey) (*types.MessageReceipt, error)
// StateMinerSectorCount returns the number of sectors in a miner's sector set and proving set
StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (MinerSectors, error)
// StateCompute is a flexible command that applies the given messages on the given tipset.
// The messages are run as though the VM were at the provided height.
StateCompute(context.Context, abi.ChainEpoch, []*types.Message, types.TipSetKey) (*ComputeStateOutput, error)
//
// When called, StateCompute will:
// - Load the provided tipset, or use the current chain head if not provided
// - Compute the tipset state of the provided tipset on top of the parent state
// - (note that this step runs before vmheight is applied to the execution)
// - Execute state upgrade if any were scheduled at the epoch, or in null
// blocks preceding the tipset
// - Call the cron actor on null blocks preceding the tipset
// - For each block in the tipset
// - Apply messages in blocks in the specified
// - Award block reward by calling the reward actor
// - Call the cron actor for the current epoch
// - If the specified vmheight is higher than the current epoch, apply any
// needed state upgrades to the state
// - Apply the specified messages to the state
//
// The vmheight parameter sets VM execution epoch, and can be used to simulate
// message execution in different network versions. If the specified vmheight
// epoch is higher than the epoch of the specified tipset, any state upgrades
// until the vmheight will be executed on the state before applying messages
// specified by the user.
//
// Note that the initial tipset state computation is not affected by the
// vmheight parameter - only the messages in the `apply` set are
//
// If the caller wants to simply compute the state, vmheight should be set to
// the epoch of the specified tipset.
//
// Messages in the `apply` parameter must have the correct nonces, and gas
// values set.
StateCompute(ctx context.Context, vmheight abi.ChainEpoch, apply []*types.Message, tsk types.TipSetKey) (*ComputeStateOutput, error)
// StateVerifierStatus returns the data cap for the given address.
// Returns nil if there is no entry in the data cap table for the
// address.

3
api/api_storage.go
View File

@ -238,6 +238,9 @@ type AddressConfig struct {
PreCommitControl []address.Address
CommitControl []address.Address
TerminateControl []address.Address
DisableOwnerFallback bool
DisableWorkerFallback bool
}
// PendingDealInfo has info about pending deals and when they are due to be

12
api/api_test.go
View File

@ -37,6 +37,18 @@ func TestDoesntDependOnFFI(t *testing.T) {
}
}
func TestDoesntDependOnBuild(t *testing.T) {
deps, err := exec.Command(goCmd(), "list", "-deps", "github.com/filecoin-project/lotus/api").Output()
if err != nil {
t.Fatal(err)
}
for _, pkg := range strings.Fields(string(deps)) {
if pkg == "github.com/filecoin-project/build" {
t.Fatal("api depends on filecoin-ffi")
}
}
}
func TestReturnTypes(t *testing.T) {
errType := reflect.TypeOf(new(error)).Elem()
bareIface := reflect.TypeOf(new(interface{})).Elem()

4
api/api_worker.go
View File

@ -9,12 +9,10 @@ import (
"github.com/filecoin-project/lotus/extern/sector-storage/sealtasks"
"github.com/filecoin-project/lotus/extern/sector-storage/stores"
"github.com/filecoin-project/lotus/extern/sector-storage/storiface"
"github.com/filecoin-project/lotus/build"
)
type WorkerAPI interface {
Version(context.Context) (build.Version, error)
Version(context.Context) (Version, error)
// TODO: Info() (name, ...) ?
TaskTypes(context.Context) (map[sealtasks.TaskType]struct{}, error) // TaskType -> Weight

38
api/apistruct/struct.go
View File

@ -5,6 +5,8 @@ import (
"io"
"time"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
"github.com/google/uuid"
"github.com/ipfs/go-cid"
metrics "github.com/libp2p/go-libp2p-core/metrics"
@ -12,8 +14,6 @@ import (
"github.com/libp2p/go-libp2p-core/peer"
protocol "github.com/libp2p/go-libp2p-core/protocol"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-fil-markets/piecestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
@ -33,7 +33,7 @@ import (
"github.com/filecoin-project/specs-storage/storage"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
apitypes "github.com/filecoin-project/lotus/api/types"
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
"github.com/filecoin-project/lotus/chain/actors/builtin/paych"
"github.com/filecoin-project/lotus/chain/types"
@ -60,12 +60,14 @@ type CommonStruct struct {
NetBandwidthStatsByPeer func(ctx context.Context) (map[string]metrics.Stats, error) `perm:"read"`
NetBandwidthStatsByProtocol func(ctx context.Context) (map[protocol.ID]metrics.Stats, error) `perm:"read"`
NetAgentVersion func(ctx context.Context, p peer.ID) (string, error) `perm:"read"`
NetPeerInfo func(context.Context, peer.ID) (*api.ExtendedPeerInfo, error) `perm:"read"`
NetBlockAdd func(ctx context.Context, acl api.NetBlockList) error `perm:"admin"`
NetBlockRemove func(ctx context.Context, acl api.NetBlockList) error `perm:"admin"`
NetBlockList func(ctx context.Context) (api.NetBlockList, error) `perm:"read"`
Discover func(ctx context.Context) (map[string]interface{}, error) `perm:"read"`
ID func(context.Context) (peer.ID, error) `perm:"read"`
Version func(context.Context) (api.Version, error) `perm:"read"`
ID func(context.Context) (peer.ID, error) `perm:"read"`
Version func(context.Context) (api.APIVersion, error) `perm:"read"`
LogList func(context.Context) ([]string, error) `perm:"write"`
LogSetLevel func(context.Context, string, string) error `perm:"write"`
@ -382,6 +384,8 @@ type StorageMinerStruct struct {
CreateBackup func(ctx context.Context, fpath string) error `perm:"admin"`
CheckProvable func(ctx context.Context, pp abi.RegisteredPoStProof, sectors []storage.SectorRef, expensive bool) (map[abi.SectorNumber]string, error) `perm:"admin"`
Discover func(ctx context.Context) (apitypes.OpenRPCDocument, error) `perm:"read"`
}
}
@ -389,7 +393,7 @@ type WorkerStruct struct {
Internal struct {
// TODO: lower perms
Version func(context.Context) (build.Version, error) `perm:"admin"`
Version func(context.Context) (api.Version, error) `perm:"admin"`
TaskTypes func(context.Context) (map[sealtasks.TaskType]struct{}, error) `perm:"admin"`
Paths func(context.Context) ([]stores.StoragePath, error) `perm:"admin"`
@ -420,6 +424,8 @@ type WorkerStruct struct {
ProcessSession func(context.Context) (uuid.UUID, error) `perm:"admin"`
Session func(context.Context) (uuid.UUID, error) `perm:"admin"`
Discover func(ctx context.Context) (apitypes.OpenRPCDocument, error) `perm:"read"`
}
}
@ -540,13 +546,21 @@ func (c *CommonStruct) NetAgentVersion(ctx context.Context, p peer.ID) (string,
return c.Internal.NetAgentVersion(ctx, p)
}
func (c *CommonStruct) NetPeerInfo(ctx context.Context, p peer.ID) (*api.ExtendedPeerInfo, error) {
return c.Internal.NetPeerInfo(ctx, p)
}
func (c *CommonStruct) Discover(ctx context.Context) (apitypes.OpenRPCDocument, error) {
return c.Internal.Discover(ctx)
}
// ID implements API.ID
func (c *CommonStruct) ID(ctx context.Context) (peer.ID, error) {
return c.Internal.ID(ctx)
}
// Version implements API.Version
func (c *CommonStruct) Version(ctx context.Context) (api.Version, error) {
func (c *CommonStruct) Version(ctx context.Context) (api.APIVersion, error) {
return c.Internal.Version(ctx)
}
@ -1608,9 +1622,13 @@ func (c *StorageMinerStruct) CheckProvable(ctx context.Context, pp abi.Registere
return c.Internal.CheckProvable(ctx, pp, sectors, expensive)
}
func (c *StorageMinerStruct) Discover(ctx context.Context) (apitypes.OpenRPCDocument, error) {
return c.Internal.Discover(ctx)
}
// WorkerStruct
func (w *WorkerStruct) Version(ctx context.Context) (build.Version, error) {
func (w *WorkerStruct) Version(ctx context.Context) (api.Version, error) {
return w.Internal.Version(ctx)
}
@ -1706,6 +1724,10 @@ func (w *WorkerStruct) Session(ctx context.Context) (uuid.UUID, error) {
return w.Internal.Session(ctx)
}
func (c *WorkerStruct) Discover(ctx context.Context) (apitypes.OpenRPCDocument, error) {
return c.Internal.Discover(ctx)
}
func (g GatewayStruct) ChainGetBlockMessages(ctx context.Context, c cid.Cid) (*api.BlockMessages, error) {
return g.Internal.ChainGetBlockMessages(ctx, c)
}

77
api/docgen-openrpc/cmd/docgen_openrpc.go Normal file
View File

@ -0,0 +1,77 @@
package main
import (
"compress/gzip"
"encoding/json"
"io"
"log"
"os"
"github.com/filecoin-project/lotus/api/apistruct"
docgen_openrpc "github.com/filecoin-project/lotus/api/docgen-openrpc"
)
/*
main defines a small program that writes an OpenRPC document describing
a Lotus API to stdout.
If the first argument is "miner", the document will describe the StorageMiner API.
If not (no, or any other args), the document will describe the Full API.
Use:
go run ./api/openrpc/cmd ["api/api_full.go"|"api/api_storage.go"|"api/api_worker.go"] ["FullNode"|"StorageMiner"|"WorkerAPI"]
With gzip compression: a '-gzip' flag is made available as an optional third argument. Note that position matters.
go run ./api/openrpc/cmd ["api/api_full.go"|"api/api_storage.go"|"api/api_worker.go"] ["FullNode"|"StorageMiner"|"WorkerAPI"] -gzip
*/
func main() {
doc := docgen_openrpc.NewLotusOpenRPCDocument()
switch os.Args[2] {
case "FullNode":
doc.RegisterReceiverName("Filecoin", &apistruct.FullNodeStruct{})
case "StorageMiner":
doc.RegisterReceiverName("Filecoin", &apistruct.StorageMinerStruct{})
case "WorkerAPI":
doc.RegisterReceiverName("Filecoin", &apistruct.WorkerStruct{})
}
out, err := doc.Discover()
if err != nil {
log.Fatalln(err)
}
var jsonOut []byte
var writer io.WriteCloser
// Use os.Args to handle a somewhat hacky flag for the gzip option.
// Could use flags package to handle this more cleanly, but that requires changes elsewhere
// the scope of which just isn't warranted by this one use case which will usually be run
// programmatically anyways.
if len(os.Args) > 3 && os.Args[3] == "-gzip" {
jsonOut, err = json.Marshal(out)
if err != nil {
log.Fatalln(err)
}
writer = gzip.NewWriter(os.Stdout)
} else {
jsonOut, err = json.MarshalIndent(out, "", " ")
if err != nil {
log.Fatalln(err)
}
writer = os.Stdout
}
_, err = writer.Write(jsonOut)
if err != nil {
log.Fatalln(err)
}
err = writer.Close()
if err != nil {
log.Fatalln(err)
}
}

172
api/docgen-openrpc/openrpc.go Normal file
View File

@ -0,0 +1,172 @@
package docgenopenrpc
import (
"encoding/json"
"go/ast"
"net"
"os"
"reflect"
"github.com/alecthomas/jsonschema"
go_openrpc_reflect "github.com/etclabscore/go-openrpc-reflect"
"github.com/filecoin-project/lotus/api/docgen"
"github.com/filecoin-project/lotus/build"
"github.com/ipfs/go-cid"
meta_schema "github.com/open-rpc/meta-schema"
)
// Comments holds API method comments collected by AST parsing.
var Comments map[string]string
// GroupDocs holds documentation for documentation groups.
var GroupDocs map[string]string
func init() {
Comments, GroupDocs = docgen.ParseApiASTInfo(os.Args[1], os.Args[2])
}
// schemaDictEntry represents a type association passed to the jsonschema reflector.
type schemaDictEntry struct {
example interface{}
rawJson string
}
const integerD = `{
"title": "number",
"type": "number",
"description": "Number is a number"
}`
const cidCidD = `{"title": "Content Identifier", "type": "string", "description": "Cid represents a self-describing content addressed identifier. It is formed by a Version, a Codec (which indicates a multicodec-packed content type) and a Multihash."}`
func OpenRPCSchemaTypeMapper(ty reflect.Type) *jsonschema.Type {
unmarshalJSONToJSONSchemaType := func(input string) *jsonschema.Type {
var js jsonschema.Type
err := json.Unmarshal([]byte(input), &js)
if err != nil {
panic(err)
}
return &js
}
if ty.Kind() == reflect.Ptr {
ty = ty.Elem()
}
if ty == reflect.TypeOf((*interface{})(nil)).Elem() {
return &jsonschema.Type{Type: "object", AdditionalProperties: []byte("true")}
}
// Second, handle other types.
// Use a slice instead of a map because it preserves order, as a logic safeguard/fallback.
dict := []schemaDictEntry{
{cid.Cid{}, cidCidD},
}
for _, d := range dict {
if reflect.TypeOf(d.example) == ty {
tt := unmarshalJSONToJSONSchemaType(d.rawJson)
return tt
}
}
// Handle primitive types in case there are generic cases
// specific to our services.
switch ty.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
// Return all integer types as the hex representation integer schemea.
ret := unmarshalJSONToJSONSchemaType(integerD)
return ret
case reflect.Uintptr:
return &jsonschema.Type{Type: "number", Title: "uintptr-title"}
case reflect.Struct:
case reflect.Map:
case reflect.Slice, reflect.Array:
case reflect.Float32, reflect.Float64:
case reflect.Bool:
case reflect.String:
case reflect.Ptr, reflect.Interface:
default:
}
return nil
}
// NewLotusOpenRPCDocument defines application-specific documentation and configuration for its OpenRPC document.
func NewLotusOpenRPCDocument() *go_openrpc_reflect.Document {
d := &go_openrpc_reflect.Document{}
// Register "Meta" document fields.
// These include getters for
// - Servers object
// - Info object
// - ExternalDocs object
//
// These objects represent server-specific data that cannot be
// reflected.
d.WithMeta(&go_openrpc_reflect.MetaT{
GetServersFn: func() func(listeners []net.Listener) (*meta_schema.Servers, error) {
return func(listeners []net.Listener) (*meta_schema.Servers, error) {
return nil, nil
}
},
GetInfoFn: func() (info *meta_schema.InfoObject) {
info = &meta_schema.InfoObject{}
title := "Lotus RPC API"
info.Title = (*meta_schema.InfoObjectProperties)(&title)
version := build.BuildVersion
info.Version = (*meta_schema.InfoObjectVersion)(&version)
return info
},
GetExternalDocsFn: func() (exdocs *meta_schema.ExternalDocumentationObject) {
return nil // FIXME
},
})
// Use a provided Ethereum default configuration as a base.
appReflector := &go_openrpc_reflect.EthereumReflectorT{}
// Install overrides for the json schema->type map fn used by the jsonschema reflect package.
appReflector.FnSchemaTypeMap = func() func(ty reflect.Type) *jsonschema.Type {
return OpenRPCSchemaTypeMapper
}
appReflector.FnIsMethodEligible = func(m reflect.Method) bool {
for i := 0; i < m.Func.Type().NumOut(); i++ {
if m.Func.Type().Out(i).Kind() == reflect.Chan {
return false
}
}
return go_openrpc_reflect.EthereumReflector.IsMethodEligible(m)
}
appReflector.FnGetMethodName = func(moduleName string, r reflect.Value, m reflect.Method, funcDecl *ast.FuncDecl) (string, error) {
if m.Name == "ID" {
return moduleName + "_ID", nil
}
if moduleName == "rpc" && m.Name == "Discover" {
return "rpc.discover", nil
}
return moduleName + "." + m.Name, nil
}
appReflector.FnGetMethodSummary = func(r reflect.Value, m reflect.Method, funcDecl *ast.FuncDecl) (string, error) {
if v, ok := Comments[m.Name]; ok {
return v, nil
}
return "", nil // noComment
}
appReflector.FnSchemaExamples = func(ty reflect.Type) (examples *meta_schema.Examples, err error) {
v := docgen.ExampleValue("unknown", ty, ty) // This isn't ideal, but seems to work well enough.
return &meta_schema.Examples{
meta_schema.AlwaysTrue(v),
}, nil
}
// Finally, register the configured reflector to the document.
d.WithReflector(appReflector)
return d
}

137
api/docgen/cmd/docgen.go Normal file
View File

@ -0,0 +1,137 @@
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
"sort"
"strings"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/apistruct"
"github.com/filecoin-project/lotus/api/docgen"
)
func main() {
comments, groupComments := docgen.ParseApiASTInfo(os.Args[1], os.Args[2])
groups := make(map[string]*docgen.MethodGroup)
var t reflect.Type
var permStruct, commonPermStruct reflect.Type
switch os.Args[2] {
case "FullNode":
t = reflect.TypeOf(new(struct{ api.FullNode })).Elem()
permStruct = reflect.TypeOf(apistruct.FullNodeStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.CommonStruct{}.Internal)
case "StorageMiner":
t = reflect.TypeOf(new(struct{ api.StorageMiner })).Elem()
permStruct = reflect.TypeOf(apistruct.StorageMinerStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.CommonStruct{}.Internal)
case "WorkerAPI":
t = reflect.TypeOf(new(struct{ api.WorkerAPI })).Elem()
permStruct = reflect.TypeOf(apistruct.WorkerStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.WorkerStruct{}.Internal)
default:
panic("unknown type")
}
for i := 0; i < t.NumMethod(); i++ {
m := t.Method(i)
groupName := docgen.MethodGroupFromName(m.Name)
g, ok := groups[groupName]
if !ok {
g = new(docgen.MethodGroup)
g.Header = groupComments[groupName]
g.GroupName = groupName
groups[groupName] = g
}
var args []interface{}
ft := m.Func.Type()
for j := 2; j < ft.NumIn(); j++ {
inp := ft.In(j)
args = append(args, docgen.ExampleValue(m.Name, inp, nil))
}
v, err := json.MarshalIndent(args, "", " ")
if err != nil {
panic(err)
}
outv := docgen.ExampleValue(m.Name, ft.Out(0), nil)
ov, err := json.MarshalIndent(outv, "", " ")
if err != nil {
panic(err)
}
g.Methods = append(g.Methods, &docgen.Method{
Name: m.Name,
Comment: comments[m.Name],
InputExample: string(v),
ResponseExample: string(ov),
})
}
var groupslice []*docgen.MethodGroup
for _, g := range groups {
groupslice = append(groupslice, g)
}
sort.Slice(groupslice, func(i, j int) bool {
return groupslice[i].GroupName < groupslice[j].GroupName
})
fmt.Printf("# Groups\n")
for _, g := range groupslice {
fmt.Printf("* [%s](#%s)\n", g.GroupName, g.GroupName)
for _, method := range g.Methods {
fmt.Printf(" * [%s](#%s)\n", method.Name, method.Name)
}
}
for _, g := range groupslice {
g := g
fmt.Printf("## %s\n", g.GroupName)
fmt.Printf("%s\n\n", g.Header)
sort.Slice(g.Methods, func(i, j int) bool {
return g.Methods[i].Name < g.Methods[j].Name
})
for _, m := range g.Methods {
fmt.Printf("### %s\n", m.Name)
fmt.Printf("%s\n\n", m.Comment)
meth, ok := permStruct.FieldByName(m.Name)
if !ok {
meth, ok = commonPermStruct.FieldByName(m.Name)
if !ok {
panic("no perms for method: " + m.Name)
}
}
perms := meth.Tag.Get("perm")
fmt.Printf("Perms: %s\n\n", perms)
if strings.Count(m.InputExample, "\n") > 0 {
fmt.Printf("Inputs:\n```json\n%s\n```\n\n", m.InputExample)
} else {
fmt.Printf("Inputs: `%s`\n\n", m.InputExample)
}
if strings.Count(m.ResponseExample, "\n") > 0 {
fmt.Printf("Response:\n```json\n%s\n```\n\n", m.ResponseExample)
} else {
fmt.Printf("Response: `%s`\n\n", m.ResponseExample)
}
}
}
}

211
api/docgen/docgen.go
View File

@ -1,18 +1,18 @@
package main
package docgen
import (
"encoding/json"
"fmt"
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"time"
"unicode"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
"github.com/google/uuid"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-filestore"
@ -23,8 +23,6 @@ import (
pubsub "github.com/libp2p/go-libp2p-pubsub"
"github.com/multiformats/go-multiaddr"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
datatransfer "github.com/filecoin-project/go-data-transfer"
filestore2 "github.com/filecoin-project/go-fil-markets/filestore"
"github.com/filecoin-project/go-fil-markets/retrievalmarket"
@ -36,7 +34,7 @@ import (
"github.com/filecoin-project/go-state-types/exitcode"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/apistruct"
apitypes "github.com/filecoin-project/lotus/api/types"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/extern/sector-storage/sealtasks"
@ -89,6 +87,8 @@ func init() {
addExample(pid)
addExample(&pid)
multistoreIDExample := multistore.StoreID(50)
addExample(bitfield.NewFromSet([]uint64{5}))
addExample(abi.RegisteredSealProof_StackedDrg32GiBV1_1)
addExample(abi.RegisteredPoStProof_StackedDrgWindow32GiBV1)
@ -113,28 +113,31 @@ func init() {
addExample(network.Connected)
addExample(dtypes.NetworkName("lotus"))
addExample(api.SyncStateStage(1))
addExample(build.FullAPIVersion)
addExample(api.FullAPIVersion)
addExample(api.PCHInbound)
addExample(time.Minute)
addExample(datatransfer.TransferID(3))
addExample(datatransfer.Ongoing)
addExample(multistore.StoreID(50))
addExample(multistoreIDExample)
addExample(&multistoreIDExample)
addExample(retrievalmarket.ClientEventDealAccepted)
addExample(retrievalmarket.DealStatusNew)
addExample(network.ReachabilityPublic)
addExample(build.NewestNetworkVersion)
addExample(map[string]int{"name": 42})
addExample(map[string]time.Time{"name": time.Unix(1615243938, 0).UTC()})
addExample(&types.ExecutionTrace{
Msg: exampleValue("init", reflect.TypeOf(&types.Message{}), nil).(*types.Message),
MsgRct: exampleValue("init", reflect.TypeOf(&types.MessageReceipt{}), nil).(*types.MessageReceipt),
Msg: ExampleValue("init", reflect.TypeOf(&types.Message{}), nil).(*types.Message),
MsgRct: ExampleValue("init", reflect.TypeOf(&types.MessageReceipt{}), nil).(*types.MessageReceipt),
})
addExample(map[string]types.Actor{
"t01236": exampleValue("init", reflect.TypeOf(types.Actor{}), nil).(types.Actor),
"t01236": ExampleValue("init", reflect.TypeOf(types.Actor{}), nil).(types.Actor),
})
addExample(map[string]api.MarketDeal{
"t026363": exampleValue("init", reflect.TypeOf(api.MarketDeal{}), nil).(api.MarketDeal),
"t026363": ExampleValue("init", reflect.TypeOf(api.MarketDeal{}), nil).(api.MarketDeal),
})
addExample(map[string]api.MarketBalance{
"t026363": exampleValue("init", reflect.TypeOf(api.MarketBalance{}), nil).(api.MarketBalance),
"t026363": ExampleValue("init", reflect.TypeOf(api.MarketBalance{}), nil).(api.MarketBalance),
})
addExample(map[string]*pubsub.TopicScoreSnapshot{
"/blocks": {
@ -249,9 +252,17 @@ func init() {
sealtasks.TTPreCommit2: {},
})
addExample(sealtasks.TTCommit2)
addExample(apitypes.OpenRPCDocument{
"openrpc": "1.2.6",
"info": map[string]interface{}{
"title": "Lotus RPC API",
"version": "1.2.1/generated=2020-11-22T08:22:42-06:00",
},
"methods": []interface{}{}},
)
}
func exampleValue(method string, t, parent reflect.Type) interface{} {
func ExampleValue(method string, t, parent reflect.Type) interface{} {
v, ok := ExampleValues[t]
if ok {
return v
@ -260,10 +271,10 @@ func exampleValue(method string, t, parent reflect.Type) interface{} {
switch t.Kind() {
case reflect.Slice:
out := reflect.New(t).Elem()
reflect.Append(out, reflect.ValueOf(exampleValue(method, t.Elem(), t)))
reflect.Append(out, reflect.ValueOf(ExampleValue(method, t.Elem(), t)))
return out.Interface()
case reflect.Chan:
return exampleValue(method, t.Elem(), nil)
return ExampleValue(method, t.Elem(), nil)
case reflect.Struct:
es := exampleStruct(method, t, parent)
v := reflect.ValueOf(es).Elem().Interface()
@ -272,7 +283,7 @@ func exampleValue(method string, t, parent reflect.Type) interface{} {
case reflect.Array:
out := reflect.New(t).Elem()
for i := 0; i < t.Len(); i++ {
out.Index(i).Set(reflect.ValueOf(exampleValue(method, t.Elem(), t)))
out.Index(i).Set(reflect.ValueOf(ExampleValue(method, t.Elem(), t)))
}
return out.Interface()
@ -297,7 +308,7 @@ func exampleStruct(method string, t, parent reflect.Type) interface{} {
continue
}
if strings.Title(f.Name) == f.Name {
ns.Elem().Field(i).Set(reflect.ValueOf(exampleValue(method, f.Type, t)))
ns.Elem().Field(i).Set(reflect.ValueOf(ExampleValue(method, f.Type, t)))
}
}
@ -329,13 +340,24 @@ func (v *Visitor) Visit(node ast.Node) ast.Visitor {
return v
}
const noComment = "There are not yet any comments for this method."
const NoComment = "There are not yet any comments for this method."
func parseApiASTInfo(apiFile, iface string) (map[string]string, map[string]string) { //nolint:golint
func ParseApiASTInfo(apiFile, iface string) (comments map[string]string, groupDocs map[string]string) { //nolint:golint
fset := token.NewFileSet()
pkgs, err := parser.ParseDir(fset, "./api", nil, parser.AllErrors|parser.ParseComments)
apiDir, err := filepath.Abs("./api")
if err != nil {
fmt.Println("./api filepath absolute error: ", err)
return
}
apiFile, err = filepath.Abs(apiFile)
if err != nil {
fmt.Println("filepath absolute error: ", err, "file:", apiFile)
return
}
pkgs, err := parser.ParseDir(fset, apiDir, nil, parser.AllErrors|parser.ParseComments)
if err != nil {
fmt.Println("parse error: ", err)
return
}
ap := pkgs["api"]
@ -347,14 +369,14 @@ func parseApiASTInfo(apiFile, iface string) (map[string]string, map[string]strin
v := &Visitor{iface, make(map[string]ast.Node)}
ast.Walk(v, pkgs["api"])
groupDocs := make(map[string]string)
out := make(map[string]string)
comments = make(map[string]string)
groupDocs = make(map[string]string)
for mn, node := range v.Methods {
cs := cmap.Filter(node).Comments()
if len(cs) == 0 {
out[mn] = noComment
filteredComments := cmap.Filter(node).Comments()
if len(filteredComments) == 0 {
comments[mn] = NoComment
} else {
for _, c := range cs {
for _, c := range filteredComments {
if strings.HasPrefix(c.Text(), "MethodGroup:") {
parts := strings.Split(c.Text(), "\n")
groupName := strings.TrimSpace(parts[0][12:])
@ -365,15 +387,15 @@ func parseApiASTInfo(apiFile, iface string) (map[string]string, map[string]strin
}
}
last := cs[len(cs)-1].Text()
last := filteredComments[len(filteredComments)-1].Text()
if !strings.HasPrefix(last, "MethodGroup:") {
out[mn] = last
comments[mn] = last
} else {
out[mn] = noComment
comments[mn] = NoComment
}
}
}
return out, groupDocs
return comments, groupDocs
}
type MethodGroup struct {
@ -389,7 +411,7 @@ type Method struct {
ResponseExample string
}
func methodGroupFromName(mn string) string {
func MethodGroupFromName(mn string) string {
i := strings.IndexFunc(mn[1:], func(r rune) bool {
return unicode.IsUpper(r)
})
@ -398,126 +420,3 @@ func methodGroupFromName(mn string) string {
}
return mn[:i+1]
}
func main() {
comments, groupComments := parseApiASTInfo(os.Args[1], os.Args[2])
groups := make(map[string]*MethodGroup)
var t reflect.Type
var permStruct, commonPermStruct reflect.Type
switch os.Args[2] {
case "FullNode":
t = reflect.TypeOf(new(struct{ api.FullNode })).Elem()
permStruct = reflect.TypeOf(apistruct.FullNodeStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.CommonStruct{}.Internal)
case "StorageMiner":
t = reflect.TypeOf(new(struct{ api.StorageMiner })).Elem()
permStruct = reflect.TypeOf(apistruct.StorageMinerStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.CommonStruct{}.Internal)
case "WorkerAPI":
t = reflect.TypeOf(new(struct{ api.WorkerAPI })).Elem()
permStruct = reflect.TypeOf(apistruct.WorkerStruct{}.Internal)
commonPermStruct = reflect.TypeOf(apistruct.WorkerStruct{}.Internal)
default:
panic("unknown type")
}
for i := 0; i < t.NumMethod(); i++ {
m := t.Method(i)
groupName := methodGroupFromName(m.Name)
g, ok := groups[groupName]
if !ok {
g = new(MethodGroup)
g.Header = groupComments[groupName]
g.GroupName = groupName
groups[groupName] = g
}
var args []interface{}
ft := m.Func.Type()
for j := 2; j < ft.NumIn(); j++ {
inp := ft.In(j)
args = append(args, exampleValue(m.Name, inp, nil))
}
v, err := json.MarshalIndent(args, "", " ")
if err != nil {
panic(err)
}
outv := exampleValue(m.Name, ft.Out(0), nil)
ov, err := json.MarshalIndent(outv, "", " ")
if err != nil {
panic(err)
}
g.Methods = append(g.Methods, &Method{
Name: m.Name,
Comment: comments[m.Name],
InputExample: string(v),
ResponseExample: string(ov),
})
}
var groupslice []*MethodGroup
for _, g := range groups {
groupslice = append(groupslice, g)
}
sort.Slice(groupslice, func(i, j int) bool {
return groupslice[i].GroupName < groupslice[j].GroupName
})
fmt.Printf("# Groups\n")
for _, g := range groupslice {
fmt.Printf("* [%s](#%s)\n", g.GroupName, g.GroupName)
for _, method := range g.Methods {
fmt.Printf(" * [%s](#%s)\n", method.Name, method.Name)
}
}
for _, g := range groupslice {
g := g
fmt.Printf("## %s\n", g.GroupName)
fmt.Printf("%s\n\n", g.Header)
sort.Slice(g.Methods, func(i, j int) bool {
return g.Methods[i].Name < g.Methods[j].Name
})
for _, m := range g.Methods {
fmt.Printf("### %s\n", m.Name)
fmt.Printf("%s\n\n", m.Comment)
meth, ok := permStruct.FieldByName(m.Name)
if !ok {
meth, ok = commonPermStruct.FieldByName(m.Name)
if !ok {
panic("no perms for method: " + m.Name)
}
}
perms := meth.Tag.Get("perm")
fmt.Printf("Perms: %s\n\n", perms)
if strings.Count(m.InputExample, "\n") > 0 {
fmt.Printf("Inputs:\n```json\n%s\n```\n\n", m.InputExample)
} else {
fmt.Printf("Inputs: `%s`\n\n", m.InputExample)
}
if strings.Count(m.ResponseExample, "\n") > 0 {
fmt.Printf("Response:\n```json\n%s\n```\n\n", m.ResponseExample)
} else {
fmt.Printf("Response: `%s`\n\n", m.ResponseExample)
}
}
}
}

35
api/mocks/mock_full.go
View File

@ -18,6 +18,7 @@ import (
dline "github.com/filecoin-project/go-state-types/dline"
network "github.com/filecoin-project/go-state-types/network"
api "github.com/filecoin-project/lotus/api"
apitypes "github.com/filecoin-project/lotus/api/types"
miner "github.com/filecoin-project/lotus/chain/actors/builtin/miner"
types "github.com/filecoin-project/lotus/chain/types"
marketevents "github.com/filecoin-project/lotus/markets/loggers"
@ -783,6 +784,21 @@ func (mr *MockFullNodeMockRecorder) CreateBackup(arg0, arg1 interface{}) *gomock
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateBackup", reflect.TypeOf((*MockFullNode)(nil).CreateBackup), arg0, arg1)
}
// Discover mocks base method
func (m *MockFullNode) Discover(arg0 context.Context) (apitypes.OpenRPCDocument, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Discover", arg0)
ret0, _ := ret[0].(apitypes.OpenRPCDocument)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// Discover indicates an expected call of Discover
func (mr *MockFullNodeMockRecorder) Discover(arg0 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Discover", reflect.TypeOf((*MockFullNode)(nil).Discover), arg0)
}
// GasEstimateFeeCap mocks base method
func (m *MockFullNode) GasEstimateFeeCap(arg0 context.Context, arg1 *types.Message, arg2 int64, arg3 types.TipSetKey) (big.Int, error) {
m.ctrl.T.Helper()
@ -1615,6 +1631,21 @@ func (mr *MockFullNodeMockRecorder) NetFindPeer(arg0, arg1 interface{}) *gomock.
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NetFindPeer", reflect.TypeOf((*MockFullNode)(nil).NetFindPeer), arg0, arg1)
}
// NetPeerInfo mocks base method
func (m *MockFullNode) NetPeerInfo(arg0 context.Context, arg1 peer.ID) (*api.ExtendedPeerInfo, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "NetPeerInfo", arg0, arg1)
ret0, _ := ret[0].(*api.ExtendedPeerInfo)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// NetPeerInfo indicates an expected call of NetPeerInfo
func (mr *MockFullNodeMockRecorder) NetPeerInfo(arg0, arg1 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NetPeerInfo", reflect.TypeOf((*MockFullNode)(nil).NetPeerInfo), arg0, arg1)
}
// NetPeers mocks base method
func (m *MockFullNode) NetPeers(arg0 context.Context) ([]peer.AddrInfo, error) {
m.ctrl.T.Helper()
@ -2764,10 +2795,10 @@ func (mr *MockFullNodeMockRecorder) SyncValidateTipset(arg0, arg1 interface{}) *
}
// Version mocks base method
func (m *MockFullNode) Version(arg0 context.Context) (api.Version, error) {
func (m *MockFullNode) Version(arg0 context.Context) (api.APIVersion, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Version", arg0)
ret0, _ := ret[0].(api.Version)
ret0, _ := ret[0].(api.APIVersion)
ret1, _ := ret[1].(error)
return ret0, ret1
}

67
api/test/deals.go
View File

@ -23,9 +23,11 @@ import (
"github.com/filecoin-project/lotus/chain/actors/builtin/market"
"github.com/filecoin-project/lotus/chain/types"
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
"github.com/filecoin-project/lotus/extern/storage-sealing/sealiface"
"github.com/filecoin-project/lotus/markets/storageadapter"
"github.com/filecoin-project/lotus/node"
"github.com/filecoin-project/lotus/node/impl"
"github.com/filecoin-project/lotus/node/modules/dtypes"
market2 "github.com/filecoin-project/specs-actors/v2/actors/builtin/market"
ipld "github.com/ipfs/go-ipld-format"
dag "github.com/ipfs/go-merkledag"
@ -183,6 +185,71 @@ func TestPublishDealsBatching(t *testing.T, b APIBuilder, blocktime time.Duratio
}
}
func TestBatchDealInput(t *testing.T, b APIBuilder, blocktime time.Duration, startEpoch abi.ChainEpoch) {
publishPeriod := 10 * time.Second
maxDealsPerMsg := uint64(4)
// Set max deals per publish deals message to maxDealsPerMsg
minerDef := []StorageMiner{{
Full: 0,
Opts: node.Options(
node.Override(
new(*storageadapter.DealPublisher),
storageadapter.NewDealPublisher(nil, storageadapter.PublishMsgConfig{
Period: publishPeriod,
MaxDealsPerMsg: maxDealsPerMsg,
})),
node.Override(new(dtypes.GetSealingConfigFunc), func() (dtypes.GetSealingConfigFunc, error) {
return func() (sealiface.Config, error) {
return sealiface.Config{
MaxWaitDealsSectors: 1,
MaxSealingSectors: 1,
MaxSealingSectorsForDeals: 2,
AlwaysKeepUnsealedCopy: true,
}, nil
}, nil
}),
),
Preseal: PresealGenesis,
}}
// Create a connect client and miner node
n, sn := b(t, OneFull, minerDef)
client := n[0].FullNode.(*impl.FullNodeAPI)
miner := sn[0]
s := connectAndStartMining(t, b, blocktime, client, miner)
defer s.blockMiner.Stop()
// Starts a deal and waits until it's published
runDealTillSeal := func(rseed int) {
res, _, err := CreateClientFile(s.ctx, s.client, rseed)
require.NoError(t, err)
dc := startDeal(t, s.ctx, s.miner, s.client, res.Root, false, startEpoch)
waitDealSealed(t, s.ctx, s.miner, s.client, dc, false)
}
// Run maxDealsPerMsg+1 deals in parallel
done := make(chan struct{}, maxDealsPerMsg+1)
for rseed := 1; rseed <= int(maxDealsPerMsg+1); rseed++ {
rseed := rseed
go func() {
runDealTillSeal(rseed)
done <- struct{}{}
}()
}
// Wait for maxDealsPerMsg of the deals to be published
for i := 0; i < int(maxDealsPerMsg); i++ {
<-done
}
sl, err := sn[0].SectorsList(s.ctx)
require.NoError(t, err)
require.GreaterOrEqual(t, len(sl), 4)
require.LessOrEqual(t, len(sl), 5)
}
func TestFastRetrievalDealFlow(t *testing.T, b APIBuilder, blocktime time.Duration, startEpoch abi.ChainEpoch) {
s := setupOneClientOneMiner(t, b, blocktime)
defer s.blockMiner.Stop()

6
api/test/test.go
View File

@ -155,13 +155,13 @@ var MineNext = miner.MineReq{
}
func (ts *testSuite) testVersion(t *testing.T) {
build.RunningNodeType = build.NodeFull
api.RunningNodeType = api.NodeFull
ctx := context.Background()
apis, _ := ts.makeNodes(t, OneFull, OneMiner)
api := apis[0]
napi := apis[0]
v, err := api.Version(ctx)
v, err := napi.Version(ctx)
if err != nil {
t.Fatal(err)
}

16
api/types.go
View File

@ -3,6 +3,7 @@ package api
import (
"encoding/json"
"fmt"
"time"
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-state-types/abi"
@ -101,3 +102,18 @@ type NetBlockList struct {
IPAddrs []string
IPSubnets []string
}
type ExtendedPeerInfo struct {
ID peer.ID
Agent string
Addrs []string
Protocols []string
ConnMgrMeta *ConnMgrInfo
}
type ConnMgrInfo struct {
FirstSeen time.Time
Value int
Tags map[string]int
Conns map[string]time.Time
}

3
api/types/openrpc.go Normal file
View File

@ -0,0 +1,3 @@
package apitypes
type OpenRPCDocument map[string]interface{}

71
api/version.go Normal file
View File

@ -0,0 +1,71 @@
package api
import (
"fmt"
xerrors "golang.org/x/xerrors"
)
type Version uint32
func newVer(major, minor, patch uint8) Version {
return Version(uint32(major)<<16 | uint32(minor)<<8 | uint32(patch))
}
// Ints returns (major, minor, patch) versions
func (ve Version) Ints() (uint32, uint32, uint32) {
v := uint32(ve)
return (v & majorOnlyMask) >> 16, (v & minorOnlyMask) >> 8, v & patchOnlyMask
}
func (ve Version) String() string {
vmj, vmi, vp := ve.Ints()
return fmt.Sprintf("%d.%d.%d", vmj, vmi, vp)
}
func (ve Version) EqMajorMinor(v2 Version) bool {
return ve&minorMask == v2&minorMask
}
type NodeType int
const (
NodeUnknown NodeType = iota
NodeFull
NodeMiner
NodeWorker
)
var RunningNodeType NodeType
func VersionForType(nodeType NodeType) (Version, error) {
switch nodeType {
case NodeFull:
return FullAPIVersion, nil
case NodeMiner:
return MinerAPIVersion, nil
case NodeWorker:
return WorkerAPIVersion, nil
default:
return Version(0), xerrors.Errorf("unknown node type %d", nodeType)
}
}
// semver versions of the rpc api exposed
var (
FullAPIVersion = newVer(1, 1, 0)
MinerAPIVersion = newVer(1, 0, 1)
WorkerAPIVersion = newVer(1, 0, 0)
)
//nolint:varcheck,deadcode
const (
majorMask = 0xff0000
minorMask = 0xffff00
patchMask = 0xffffff
majorOnlyMask = 0xff0000
minorOnlyMask = 0x00ff00
patchOnlyMask = 0x0000ff
)

78
blockstore/badger/blockstore.go
View File

@ -4,6 +4,7 @@ import (
"context"
"fmt"
"io"
"runtime"
"sync/atomic"
"github.com/dgraph-io/badger/v2"
@ -131,6 +132,39 @@ func (b *Blockstore) Close() error {
return b.DB.Close()
}
// CollectGarbage runs garbage collection on the value log
func (b *Blockstore) CollectGarbage() error {
if atomic.LoadInt64(&b.state) != stateOpen {
return ErrBlockstoreClosed
}
var err error
for err == nil {
err = b.DB.RunValueLogGC(0.125)
}
if err == badger.ErrNoRewrite {
// not really an error in this case
return nil
}
return err
}
// Compact runs a synchronous compaction
func (b *Blockstore) Compact() error {
if atomic.LoadInt64(&b.state) != stateOpen {
return ErrBlockstoreClosed
}
nworkers := runtime.NumCPU() / 2
if nworkers < 2 {
nworkers = 2
}
return b.DB.Flatten(nworkers)
}
// View implements blockstore.Viewer, which leverages zero-copy read-only
// access to values.
func (b *Blockstore) View(cid cid.Cid, fn func([]byte) error) error {
@ -269,9 +303,6 @@ func (b *Blockstore) PutMany(blocks []blocks.Block) error {
return ErrBlockstoreClosed
}
batch := b.DB.NewWriteBatch()
defer batch.Cancel()
// toReturn tracks the byte slices to return to the pool, if we're using key
// prefixing. we can't return each slice to the pool after each Set, because
// badger holds on to the slice.
@ -285,6 +316,9 @@ func (b *Blockstore) PutMany(blocks []blocks.Block) error {
}()
}
batch := b.DB.NewWriteBatch()
defer batch.Cancel()
for _, block := range blocks {
k, pooled := b.PooledStorageKey(block.Cid())
if pooled {
@ -318,6 +352,44 @@ func (b *Blockstore) DeleteBlock(cid cid.Cid) error {
})
}
func (b *Blockstore) DeleteMany(cids []cid.Cid) error {
if atomic.LoadInt64(&b.state) != stateOpen {
return ErrBlockstoreClosed
}
// toReturn tracks the byte slices to return to the pool, if we're using key
// prefixing. we can't return each slice to the pool after each Set, because
// badger holds on to the slice.
var toReturn [][]byte
if b.prefixing {
toReturn = make([][]byte, 0, len(cids))
defer func() {
for _, b := range toReturn {
KeyPool.Put(b)
}
}()
}
batch := b.DB.NewWriteBatch()
defer batch.Cancel()
for _, cid := range cids {
k, pooled := b.PooledStorageKey(cid)
if pooled {
toReturn = append(toReturn, k)
}
if err := batch.Delete(k); err != nil {
return err
}
}
err := batch.Flush()
if err != nil {
err = fmt.Errorf("failed to delete blocks from badger blockstore: %w", err)
}
return err
}
// AllKeysChan implements Blockstore.AllKeysChan.
func (b *Blockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error) {
if atomic.LoadInt64(&b.state) != stateOpen {

33
blockstore/blockstore.go
View File

@ -1,7 +1,7 @@
package blockstore
import (
"github.com/ipfs/go-cid"
cid "github.com/ipfs/go-cid"
ds "github.com/ipfs/go-datastore"
logging "github.com/ipfs/go-log/v2"
@ -18,6 +18,7 @@ var ErrNotFound = blockstore.ErrNotFound
type Blockstore interface {
blockstore.Blockstore
blockstore.Viewer
BatchDeleter
}
// BasicBlockstore is an alias to the original IPFS Blockstore.
@ -25,13 +26,30 @@ type BasicBlockstore = blockstore.Blockstore
type Viewer = blockstore.Viewer
type BatchDeleter interface {
DeleteMany(cids []cid.Cid) error
}
// WrapIDStore wraps the underlying blockstore in an "identity" blockstore.
// The ID store filters out all puts for blocks with CIDs using the "identity"
// hash function. It also extracts inlined blocks from CIDs using the identity
// hash function and returns them on get/has, ignoring the contents of the
// blockstore.
func WrapIDStore(bstore blockstore.Blockstore) Blockstore {
return blockstore.NewIdStore(bstore).(Blockstore)
if is, ok := bstore.(*idstore); ok {
// already wrapped
return is
}
if bs, ok := bstore.(Blockstore); ok {
// we need to wrap our own because we don't want to neuter the DeleteMany method
// the underlying blockstore has implemented an (efficient) DeleteMany
return NewIDStore(bs)
}
// The underlying blockstore does not implement DeleteMany, so we need to shim it.
// This is less efficient as it'll iterate and perform single deletes.
return NewIDStore(Adapt(bstore))
}
// FromDatastore creates a new blockstore backed by the given datastore.
@ -53,6 +71,17 @@ func (a *adaptedBlockstore) View(cid cid.Cid, callback func([]byte) error) error
return callback(blk.RawData())
}
func (a *adaptedBlockstore) DeleteMany(cids []cid.Cid) error {
for _, cid := range cids {
err := a.DeleteBlock(cid)
if err != nil {
return err
}
}
return nil
}
// Adapt adapts a standard blockstore to a Lotus blockstore by
// enriching it with the extra methods that Lotus requires (e.g. View, Sync).
//

8
blockstore/buffered.go
View File

@ -96,6 +96,14 @@ func (bs *BufferedBlockstore) DeleteBlock(c cid.Cid) error {
return bs.write.DeleteBlock(c)
}
func (bs *BufferedBlockstore) DeleteMany(cids []cid.Cid) error {
if err := bs.read.DeleteMany(cids); err != nil {
return err
}
return bs.write.DeleteMany(cids)
}
func (bs *BufferedBlockstore) View(c cid.Cid, callback func([]byte) error) error {
// both stores are viewable.
if err := bs.write.View(c, callback); err == ErrNotFound {

174
blockstore/idstore.go Normal file
View File

@ -0,0 +1,174 @@
package blockstore
import (
"context"
"io"
"golang.org/x/xerrors"
blocks "github.com/ipfs/go-block-format"
cid "github.com/ipfs/go-cid"
mh "github.com/multiformats/go-multihash"
)
var _ Blockstore = (*idstore)(nil)
type idstore struct {
bs Blockstore
}
func NewIDStore(bs Blockstore) Blockstore {
return &idstore{bs: bs}
}
func decodeCid(cid cid.Cid) (inline bool, data []byte, err error) {
if cid.Prefix().MhType != mh.IDENTITY {
return false, nil, nil
}
dmh, err := mh.Decode(cid.Hash())
if err != nil {
return false, nil, err
}
if dmh.Code == mh.IDENTITY {
return true, dmh.Digest, nil
}
return false, nil, err
}
func (b *idstore) Has(cid cid.Cid) (bool, error) {
inline, _, err := decodeCid(cid)
if err != nil {
return false, xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return true, nil
}
return b.bs.Has(cid)
}
func (b *idstore) Get(cid cid.Cid) (blocks.Block, error) {
inline, data, err := decodeCid(cid)
if err != nil {
return nil, xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return blocks.NewBlockWithCid(data, cid)
}
return b.bs.Get(cid)
}
func (b *idstore) GetSize(cid cid.Cid) (int, error) {
inline, data, err := decodeCid(cid)
if err != nil {
return 0, xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return len(data), err
}
return b.bs.GetSize(cid)
}
func (b *idstore) View(cid cid.Cid, cb func([]byte) error) error {
inline, data, err := decodeCid(cid)
if err != nil {
return xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return cb(data)
}
return b.bs.View(cid, cb)
}
func (b *idstore) Put(blk blocks.Block) error {
inline, _, err := decodeCid(blk.Cid())
if err != nil {
return xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return nil
}
return b.bs.Put(blk)
}
func (b *idstore) PutMany(blks []blocks.Block) error {
toPut := make([]blocks.Block, 0, len(blks))
for _, blk := range blks {
inline, _, err := decodeCid(blk.Cid())
if err != nil {
return xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
continue
}
toPut = append(toPut, blk)
}
if len(toPut) > 0 {
return b.bs.PutMany(toPut)
}
return nil
}
func (b *idstore) DeleteBlock(cid cid.Cid) error {
inline, _, err := decodeCid(cid)
if err != nil {
return xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
return nil
}
return b.bs.DeleteBlock(cid)
}
func (b *idstore) DeleteMany(cids []cid.Cid) error {
toDelete := make([]cid.Cid, 0, len(cids))
for _, cid := range cids {
inline, _, err := decodeCid(cid)
if err != nil {
return xerrors.Errorf("error decoding Cid: %w", err)
}
if inline {
continue
}
toDelete = append(toDelete, cid)
}
if len(toDelete) > 0 {
return b.bs.DeleteMany(toDelete)
}
return nil
}
func (b *idstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error) {
return b.bs.AllKeysChan(ctx)
}
func (b *idstore) HashOnRead(enabled bool) {
b.bs.HashOnRead(enabled)
}
func (b *idstore) Close() error {
if c, ok := b.bs.(io.Closer); ok {
return c.Close()
}
return nil
}

7
blockstore/mem.go
View File

@ -20,6 +20,13 @@ func (m MemBlockstore) DeleteBlock(k cid.Cid) error {
return nil
}
func (m MemBlockstore) DeleteMany(ks []cid.Cid) error {
for _, k := range ks {
delete(m, k)
}
return nil
}
func (m MemBlockstore) Has(k cid.Cid) (bool, error) {
_, ok := m[k]
return ok, nil

38
blockstore/splitstore/markset.go Normal file
View File

@ -0,0 +1,38 @@
package splitstore
import (
"path/filepath"
"golang.org/x/xerrors"
cid "github.com/ipfs/go-cid"
)
// MarkSet is a utility to keep track of seen CID, and later query for them.
//
// * If the expected dataset is large, it can be backed by a datastore (e.g. bbolt).
// * If a probabilistic result is acceptable, it can be backed by a bloom filter (default).
type MarkSet interface {
Mark(cid.Cid) error
Has(cid.Cid) (bool, error)
Close() error
}
// markBytes is deliberately a non-nil empty byte slice for serialization.
var markBytes = []byte{}
type MarkSetEnv interface {
Create(name string, sizeHint int64) (MarkSet, error)
Close() error
}
func OpenMarkSetEnv(path string, mtype string) (MarkSetEnv, error) {
switch mtype {
case "", "bloom":
return NewBloomMarkSetEnv()
case "bolt":
return NewBoltMarkSetEnv(filepath.Join(path, "markset.bolt"))
default:
return nil, xerrors.Errorf("unknown mark set type %s", mtype)
}
}

77
blockstore/splitstore/markset_bloom.go Normal file
View File

@ -0,0 +1,77 @@
package splitstore
import (
"crypto/rand"
"crypto/sha256"
"golang.org/x/xerrors"
bbloom "github.com/ipfs/bbloom"
cid "github.com/ipfs/go-cid"
)
const (
BloomFilterMinSize = 10_000_000
BloomFilterProbability = 0.01
)
type BloomMarkSetEnv struct{}
var _ MarkSetEnv = (*BloomMarkSetEnv)(nil)
type BloomMarkSet struct {
salt []byte
bf *bbloom.Bloom
}
var _ MarkSet = (*BloomMarkSet)(nil)
func NewBloomMarkSetEnv() (*BloomMarkSetEnv, error) {
return &BloomMarkSetEnv{}, nil
}
func (e *BloomMarkSetEnv) Create(name string, sizeHint int64) (MarkSet, error) {
size := int64(BloomFilterMinSize)
for size < sizeHint {
size += BloomFilterMinSize
}
salt := make([]byte, 4)
_, err := rand.Read(salt)
if err != nil {
return nil, xerrors.Errorf("error reading salt: %w", err)
}
bf, err := bbloom.New(float64(size), BloomFilterProbability)
if err != nil {
return nil, xerrors.Errorf("error creating bloom filter: %w", err)
}
return &BloomMarkSet{salt: salt, bf: bf}, nil
}
func (e *BloomMarkSetEnv) Close() error {
return nil
}
func (s *BloomMarkSet) saltedKey(cid cid.Cid) []byte {
hash := cid.Hash()
key := make([]byte, len(s.salt)+len(hash))
n := copy(key, s.salt)
copy(key[n:], hash)
rehash := sha256.Sum256(key)
return rehash[:]
}
func (s *BloomMarkSet) Mark(cid cid.Cid) error {
s.bf.Add(s.saltedKey(cid))
return nil
}
func (s *BloomMarkSet) Has(cid cid.Cid) (bool, error) {
return s.bf.Has(s.saltedKey(cid)), nil
}
func (s *BloomMarkSet) Close() error {
return nil
}

81
blockstore/splitstore/markset_bolt.go Normal file
View File

@ -0,0 +1,81 @@
package splitstore
import (
"time"
"golang.org/x/xerrors"
cid "github.com/ipfs/go-cid"
bolt "go.etcd.io/bbolt"
)
type BoltMarkSetEnv struct {
db *bolt.DB
}
var _ MarkSetEnv = (*BoltMarkSetEnv)(nil)
type BoltMarkSet struct {
db *bolt.DB
bucketId []byte
}
var _ MarkSet = (*BoltMarkSet)(nil)
func NewBoltMarkSetEnv(path string) (*BoltMarkSetEnv, error) {
db, err := bolt.Open(path, 0644,
&bolt.Options{
Timeout: 1 * time.Second,
NoSync: true,
})
if err != nil {
return nil, err
}
return &BoltMarkSetEnv{db: db}, nil
}
func (e *BoltMarkSetEnv) Create(name string, hint int64) (MarkSet, error) {
bucketId := []byte(name)
err := e.db.Update(func(tx *bolt.Tx) error {
_, err := tx.CreateBucketIfNotExists(bucketId)
if err != nil {
return xerrors.Errorf("error creating bolt db bucket %s: %w", name, err)
}
return nil
})
if err != nil {
return nil, err
}
return &BoltMarkSet{db: e.db, bucketId: bucketId}, nil
}
func (e *BoltMarkSetEnv) Close() error {
return e.db.Close()
}
func (s *BoltMarkSet) Mark(cid cid.Cid) error {
return s.db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
return b.Put(cid.Hash(), markBytes)
})
}
func (s *BoltMarkSet) Has(cid cid.Cid) (result bool, err error) {
err = s.db.View(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
v := b.Get(cid.Hash())
result = v != nil
return nil
})
return result, err
}
func (s *BoltMarkSet) Close() error {
return s.db.Update(func(tx *bolt.Tx) error {
return tx.DeleteBucket(s.bucketId)
})
}

138
blockstore/splitstore/markset_test.go Normal file
View File

@ -0,0 +1,138 @@
package splitstore
import (
"io/ioutil"
"testing"
cid "github.com/ipfs/go-cid"
"github.com/multiformats/go-multihash"
)
func TestBoltMarkSet(t *testing.T) {
testMarkSet(t, "bolt")
}
func TestBloomMarkSet(t *testing.T) {
testMarkSet(t, "bloom")
}
func testMarkSet(t *testing.T, lsType string) {
t.Helper()
path, err := ioutil.TempDir("", "sweep-test.*")
if err != nil {
t.Fatal(err)
}
env, err := OpenMarkSetEnv(path, lsType)
if err != nil {
t.Fatal(err)
}
defer env.Close() //nolint:errcheck
hotSet, err := env.Create("hot", 0)
if err != nil {
t.Fatal(err)
}
coldSet, err := env.Create("cold", 0)
if err != nil {
t.Fatal(err)
}
makeCid := func(key string) cid.Cid {
h, err := multihash.Sum([]byte(key), multihash.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
return cid.NewCidV1(cid.Raw, h)
}
mustHave := func(s MarkSet, cid cid.Cid) {
has, err := s.Has(cid)
if err != nil {
t.Fatal(err)
}
if !has {
t.Fatal("mark not found")
}
}
mustNotHave := func(s MarkSet, cid cid.Cid) {
has, err := s.Has(cid)
if err != nil {
t.Fatal(err)
}
if has {
t.Fatal("unexpected mark")
}
}
k1 := makeCid("a")
k2 := makeCid("b")
k3 := makeCid("c")
k4 := makeCid("d")
hotSet.Mark(k1) //nolint
hotSet.Mark(k2) //nolint
coldSet.Mark(k3) //nolint
mustHave(hotSet, k1)
mustHave(hotSet, k2)
mustNotHave(hotSet, k3)
mustNotHave(hotSet, k4)
mustNotHave(coldSet, k1)
mustNotHave(coldSet, k2)
mustHave(coldSet, k3)
mustNotHave(coldSet, k4)
// close them and reopen to redo the dance
err = hotSet.Close()
if err != nil {
t.Fatal(err)
}
err = coldSet.Close()
if err != nil {
t.Fatal(err)
}
hotSet, err = env.Create("hot", 0)
if err != nil {
t.Fatal(err)
}
coldSet, err = env.Create("cold", 0)
if err != nil {
t.Fatal(err)
}
hotSet.Mark(k3) //nolint
hotSet.Mark(k4) //nolint
coldSet.Mark(k1) //nolint
mustNotHave(hotSet, k1)
mustNotHave(hotSet, k2)
mustHave(hotSet, k3)
mustHave(hotSet, k4)
mustHave(coldSet, k1)
mustNotHave(coldSet, k2)
mustNotHave(coldSet, k3)
mustNotHave(coldSet, k4)
err = hotSet.Close()
if err != nil {
t.Fatal(err)
}
err = coldSet.Close()
if err != nil {
t.Fatal(err)
}
}

1080
blockstore/splitstore/splitstore.go Normal file
View File

File diff suppressed because it is too large Load Diff

255
blockstore/splitstore/splitstore_test.go Normal file
View File

@ -0,0 +1,255 @@
package splitstore
import (
"context"
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/blockstore"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/types/mock"
cid "github.com/ipfs/go-cid"
datastore "github.com/ipfs/go-datastore"
dssync "github.com/ipfs/go-datastore/sync"
logging "github.com/ipfs/go-log/v2"
)
func init() {
CompactionThreshold = 5
CompactionCold = 1
CompactionBoundary = 2
logging.SetLogLevel("splitstore", "DEBUG")
}
func testSplitStore(t *testing.T, cfg *Config) {
chain := &mockChain{}
// genesis
genBlock := mock.MkBlock(nil, 0, 0)
genTs := mock.TipSet(genBlock)
chain.push(genTs)
// the myriads of stores
ds := dssync.MutexWrap(datastore.NewMapDatastore())
hot := blockstore.NewMemorySync()
cold := blockstore.NewMemorySync()
// put the genesis block to cold store
blk, err := genBlock.ToStorageBlock()
if err != nil {
t.Fatal(err)
}
err = cold.Put(blk)
if err != nil {
t.Fatal(err)
}
// open the splitstore
ss, err := Open("", ds, hot, cold, cfg)
if err != nil {
t.Fatal(err)
}
defer ss.Close() //nolint
err = ss.Start(chain)
if err != nil {
t.Fatal(err)
}
// make some tipsets, but not enough to cause compaction
mkBlock := func(curTs *types.TipSet, i int) *types.TipSet {
blk := mock.MkBlock(curTs, uint64(i), uint64(i))
sblk, err := blk.ToStorageBlock()
if err != nil {
t.Fatal(err)
}
err = ss.Put(sblk)
if err != nil {
t.Fatal(err)
}
ts := mock.TipSet(blk)
chain.push(ts)
return ts
}
mkGarbageBlock := func(curTs *types.TipSet, i int) {
blk := mock.MkBlock(curTs, uint64(i), uint64(i))
sblk, err := blk.ToStorageBlock()
if err != nil {
t.Fatal(err)
}
err = ss.Put(sblk)
if err != nil {
t.Fatal(err)
}
}
waitForCompaction := func() {
for atomic.LoadInt32(&ss.compacting) == 1 {
time.Sleep(100 * time.Millisecond)
}
}
curTs := genTs
for i := 1; i < 5; i++ {
curTs = mkBlock(curTs, i)
waitForCompaction()
}
mkGarbageBlock(genTs, 1)
// count objects in the cold and hot stores
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
countBlocks := func(bs blockstore.Blockstore) int {
count := 0
ch, err := bs.AllKeysChan(ctx)
if err != nil {
t.Fatal(err)
}
for range ch {
count++
}
return count
}
coldCnt := countBlocks(cold)
hotCnt := countBlocks(hot)
if coldCnt != 1 {
t.Errorf("expected %d blocks, but got %d", 1, coldCnt)
}
if hotCnt != 5 {
t.Errorf("expected %d blocks, but got %d", 5, hotCnt)
}
// trigger a compaction
for i := 5; i < 10; i++ {
curTs = mkBlock(curTs, i)
waitForCompaction()
}
coldCnt = countBlocks(cold)
hotCnt = countBlocks(hot)
if !cfg.EnableFullCompaction {
if coldCnt != 5 {
t.Errorf("expected %d cold blocks, but got %d", 5, coldCnt)
}
if hotCnt != 5 {
t.Errorf("expected %d hot blocks, but got %d", 5, hotCnt)
}
}
if cfg.EnableFullCompaction && !cfg.EnableGC {
if coldCnt != 3 {
t.Errorf("expected %d cold blocks, but got %d", 3, coldCnt)
}
if hotCnt != 7 {
t.Errorf("expected %d hot blocks, but got %d", 7, hotCnt)
}
}
if cfg.EnableFullCompaction && cfg.EnableGC {
if coldCnt != 2 {
t.Errorf("expected %d cold blocks, but got %d", 2, coldCnt)
}
if hotCnt != 7 {
t.Errorf("expected %d hot blocks, but got %d", 7, hotCnt)
}
}
}
func TestSplitStoreSimpleCompaction(t *testing.T) {
testSplitStore(t, &Config{TrackingStoreType: "mem"})
}
func TestSplitStoreFullCompactionWithoutGC(t *testing.T) {
testSplitStore(t, &Config{
TrackingStoreType: "mem",
EnableFullCompaction: true,
})
}
func TestSplitStoreFullCompactionWithGC(t *testing.T) {
testSplitStore(t, &Config{
TrackingStoreType: "mem",
EnableFullCompaction: true,
EnableGC: true,
})
}
type mockChain struct {
sync.Mutex
tipsets []*types.TipSet
listener func(revert []*types.TipSet, apply []*types.TipSet) error
}
func (c *mockChain) push(ts *types.TipSet) {
c.Lock()
c.tipsets = append(c.tipsets, ts)
c.Unlock()
if c.listener != nil {
err := c.listener(nil, []*types.TipSet{ts})
if err != nil {
log.Errorf("mockchain: error dispatching listener: %s", err)
}
}
}
func (c *mockChain) GetTipsetByHeight(_ context.Context, epoch abi.ChainEpoch, _ *types.TipSet, _ bool) (*types.TipSet, error) {
c.Lock()
defer c.Unlock()
iEpoch := int(epoch)
if iEpoch > len(c.tipsets) {
return nil, fmt.Errorf("bad epoch %d", epoch)
}
return c.tipsets[iEpoch-1], nil
}
func (c *mockChain) GetHeaviestTipSet() *types.TipSet {
c.Lock()
defer c.Unlock()
return c.tipsets[len(c.tipsets)-1]
}
func (c *mockChain) SubscribeHeadChanges(change func(revert []*types.TipSet, apply []*types.TipSet) error) {
c.listener = change
}
func (c *mockChain) WalkSnapshot(_ context.Context, ts *types.TipSet, epochs abi.ChainEpoch, _ bool, _ bool, f func(cid.Cid) error) error {
c.Lock()
defer c.Unlock()
start := int(ts.Height()) - 1
end := start - int(epochs)
if end < 0 {
end = -1
}
for i := start; i > end; i-- {
ts := c.tipsets[i]
for _, cid := range ts.Cids() {
err := f(cid)
if err != nil {
return err
}
}
}
return nil
}

109
blockstore/splitstore/tracking.go Normal file
View File

@ -0,0 +1,109 @@
package splitstore
import (
"path/filepath"
"sync"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-state-types/abi"
cid "github.com/ipfs/go-cid"
)
// TrackingStore is a persistent store that tracks blocks that are added
// to the hotstore, tracking the epoch at which they are written.
type TrackingStore interface {
Put(cid.Cid, abi.ChainEpoch) error
PutBatch([]cid.Cid, abi.ChainEpoch) error
Get(cid.Cid) (abi.ChainEpoch, error)
Delete(cid.Cid) error
DeleteBatch([]cid.Cid) error
ForEach(func(cid.Cid, abi.ChainEpoch) error) error
Sync() error
Close() error
}
// OpenTrackingStore opens a tracking store of the specified type in the
// specified path.
func OpenTrackingStore(path string, ttype string) (TrackingStore, error) {
switch ttype {
case "", "bolt":
return OpenBoltTrackingStore(filepath.Join(path, "tracker.bolt"))
case "mem":
return NewMemTrackingStore(), nil
default:
return nil, xerrors.Errorf("unknown tracking store type %s", ttype)
}
}
// NewMemTrackingStore creates an in-memory tracking store.
// This is only useful for test or situations where you don't want to open the
// real tracking store (eg concurrent read only access on a node's datastore)
func NewMemTrackingStore() *MemTrackingStore {
return &MemTrackingStore{tab: make(map[cid.Cid]abi.ChainEpoch)}
}
// MemTrackingStore is a simple in-memory tracking store
type MemTrackingStore struct {
sync.Mutex
tab map[cid.Cid]abi.ChainEpoch
}
var _ TrackingStore = (*MemTrackingStore)(nil)
func (s *MemTrackingStore) Put(cid cid.Cid, epoch abi.ChainEpoch) error {
s.Lock()
defer s.Unlock()
s.tab[cid] = epoch
return nil
}
func (s *MemTrackingStore) PutBatch(cids []cid.Cid, epoch abi.ChainEpoch) error {
s.Lock()
defer s.Unlock()
for _, cid := range cids {
s.tab[cid] = epoch
}
return nil
}
func (s *MemTrackingStore) Get(cid cid.Cid) (abi.ChainEpoch, error) {
s.Lock()
defer s.Unlock()
epoch, ok := s.tab[cid]
if ok {
return epoch, nil
}
return 0, xerrors.Errorf("missing tracking epoch for %s", cid)
}
func (s *MemTrackingStore) Delete(cid cid.Cid) error {
s.Lock()
defer s.Unlock()
delete(s.tab, cid)
return nil
}
func (s *MemTrackingStore) DeleteBatch(cids []cid.Cid) error {
s.Lock()
defer s.Unlock()
for _, cid := range cids {
delete(s.tab, cid)
}
return nil
}
func (s *MemTrackingStore) ForEach(f func(cid.Cid, abi.ChainEpoch) error) error {
s.Lock()
defer s.Unlock()
for cid, epoch := range s.tab {
err := f(cid, epoch)
if err != nil {
return err
}
}
return nil
}
func (s *MemTrackingStore) Sync() error { return nil }
func (s *MemTrackingStore) Close() error { return nil }

120
blockstore/splitstore/tracking_bolt.go Normal file
View File

@ -0,0 +1,120 @@
package splitstore
import (
"time"
"golang.org/x/xerrors"
cid "github.com/ipfs/go-cid"
bolt "go.etcd.io/bbolt"
"github.com/filecoin-project/go-state-types/abi"
)
type BoltTrackingStore struct {
db *bolt.DB
bucketId []byte
}
var _ TrackingStore = (*BoltTrackingStore)(nil)
func OpenBoltTrackingStore(path string) (*BoltTrackingStore, error) {
opts := &bolt.Options{
Timeout: 1 * time.Second,
NoSync: true,
}
db, err := bolt.Open(path, 0644, opts)
if err != nil {
return nil, err
}
bucketId := []byte("tracker")
err = db.Update(func(tx *bolt.Tx) error {
_, err := tx.CreateBucketIfNotExists(bucketId)
if err != nil {
return xerrors.Errorf("error creating bolt db bucket %s: %w", string(bucketId), err)
}
return nil
})
if err != nil {
_ = db.Close()
return nil, err
}
return &BoltTrackingStore{db: db, bucketId: bucketId}, nil
}
func (s *BoltTrackingStore) Put(cid cid.Cid, epoch abi.ChainEpoch) error {
val := epochToBytes(epoch)
return s.db.Batch(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
return b.Put(cid.Hash(), val)
})
}
func (s *BoltTrackingStore) PutBatch(cids []cid.Cid, epoch abi.ChainEpoch) error {
val := epochToBytes(epoch)
return s.db.Batch(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
for _, cid := range cids {
err := b.Put(cid.Hash(), val)
if err != nil {
return err
}
}
return nil
})
}
func (s *BoltTrackingStore) Get(cid cid.Cid) (epoch abi.ChainEpoch, err error) {
err = s.db.View(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
val := b.Get(cid.Hash())
if val == nil {
return xerrors.Errorf("missing tracking epoch for %s", cid)
}
epoch = bytesToEpoch(val)
return nil
})
return epoch, err
}
func (s *BoltTrackingStore) Delete(cid cid.Cid) error {
return s.db.Batch(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
return b.Delete(cid.Hash())
})
}
func (s *BoltTrackingStore) DeleteBatch(cids []cid.Cid) error {
return s.db.Batch(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
for _, cid := range cids {
err := b.Delete(cid.Hash())
if err != nil {
return xerrors.Errorf("error deleting %s", cid)
}
}
return nil
})
}
func (s *BoltTrackingStore) ForEach(f func(cid.Cid, abi.ChainEpoch) error) error {
return s.db.View(func(tx *bolt.Tx) error {
b := tx.Bucket(s.bucketId)
return b.ForEach(func(k, v []byte) error {
cid := cid.NewCidV1(cid.Raw, k)
epoch := bytesToEpoch(v)
return f(cid, epoch)
})
})
}
func (s *BoltTrackingStore) Sync() error {
return s.db.Sync()
}
func (s *BoltTrackingStore) Close() error {
return s.db.Close()
}

130
blockstore/splitstore/tracking_test.go Normal file
View File

@ -0,0 +1,130 @@
package splitstore
import (
"io/ioutil"
"testing"
cid "github.com/ipfs/go-cid"
"github.com/multiformats/go-multihash"
"github.com/filecoin-project/go-state-types/abi"
)
func TestBoltTrackingStore(t *testing.T) {
testTrackingStore(t, "bolt")
}
func testTrackingStore(t *testing.T, tsType string) {
t.Helper()
makeCid := func(key string) cid.Cid {
h, err := multihash.Sum([]byte(key), multihash.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
return cid.NewCidV1(cid.Raw, h)
}
mustHave := func(s TrackingStore, cid cid.Cid, epoch abi.ChainEpoch) {
val, err := s.Get(cid)
if err != nil {
t.Fatal(err)
}
if val != epoch {
t.Fatal("epoch mismatch")
}
}
mustNotHave := func(s TrackingStore, cid cid.Cid) {
_, err := s.Get(cid)
if err == nil {
t.Fatal("expected error")
}
}
path, err := ioutil.TempDir("", "snoop-test.*")
if err != nil {
t.Fatal(err)
}
s, err := OpenTrackingStore(path, tsType)
if err != nil {
t.Fatal(err)
}
k1 := makeCid("a")
k2 := makeCid("b")
k3 := makeCid("c")
k4 := makeCid("d")
s.Put(k1, 1) //nolint
s.Put(k2, 2) //nolint
s.Put(k3, 3) //nolint
s.Put(k4, 4) //nolint
mustHave(s, k1, 1)
mustHave(s, k2, 2)
mustHave(s, k3, 3)
mustHave(s, k4, 4)
s.Delete(k1) // nolint
s.Delete(k2) // nolint
mustNotHave(s, k1)
mustNotHave(s, k2)
mustHave(s, k3, 3)
mustHave(s, k4, 4)
s.PutBatch([]cid.Cid{k1}, 1) //nolint
s.PutBatch([]cid.Cid{k2}, 2) //nolint
mustHave(s, k1, 1)
mustHave(s, k2, 2)
mustHave(s, k3, 3)
mustHave(s, k4, 4)
allKeys := map[string]struct{}{
k1.String(): {},
k2.String(): {},
k3.String(): {},
k4.String(): {},
}
err = s.ForEach(func(k cid.Cid, _ abi.ChainEpoch) error {
_, ok := allKeys[k.String()]
if !ok {
t.Fatal("unexpected key")
}
delete(allKeys, k.String())
return nil
})
if err != nil {
t.Fatal(err)
}
if len(allKeys) != 0 {
t.Fatal("not all keys were returned")
}
// no close and reopen and ensure the keys still exist
err = s.Close()
if err != nil {
t.Fatal(err)
}
s, err = OpenTrackingStore(path, tsType)
if err != nil {
t.Fatal(err)
}
mustHave(s, k1, 1)
mustHave(s, k2, 2)
mustHave(s, k3, 3)
mustHave(s, k4, 4)
s.Close() //nolint:errcheck
}

6
blockstore/sync.go
View File

@ -26,6 +26,12 @@ func (m *SyncBlockstore) DeleteBlock(k cid.Cid) error {
return m.bs.DeleteBlock(k)
}
func (m *SyncBlockstore) DeleteMany(ks []cid.Cid) error {
m.mu.Lock()
defer m.mu.Unlock()
return m.bs.DeleteMany(ks)
}
func (m *SyncBlockstore) Has(k cid.Cid) (bool, error) {
m.mu.RLock()
defer m.mu.RUnlock()

21
blockstore/timed.go
View File

@ -103,13 +103,20 @@ func (t *TimedCacheBlockstore) PutMany(bs []blocks.Block) error {
}
func (t *TimedCacheBlockstore) View(k cid.Cid, callback func([]byte) error) error {
// The underlying blockstore is always a "mem" blockstore so there's no difference,
// from a performance perspective, between view & get. So we call Get to avoid
// calling an arbitrary callback while holding a lock.
t.mu.RLock()
defer t.mu.RUnlock()
err := t.active.View(k, callback)
block, err := t.active.Get(k)
if err == ErrNotFound {
err = t.inactive.View(k, callback)
block, err = t.inactive.Get(k)
}
return err
t.mu.RUnlock()
if err != nil {
return err
}
return callback(block.RawData())
}
func (t *TimedCacheBlockstore) Get(k cid.Cid) (blocks.Block, error) {
@ -153,6 +160,12 @@ func (t *TimedCacheBlockstore) DeleteBlock(k cid.Cid) error {
return multierr.Combine(t.active.DeleteBlock(k), t.inactive.DeleteBlock(k))
}
func (t *TimedCacheBlockstore) DeleteMany(ks []cid.Cid) error {
t.mu.Lock()
defer t.mu.Unlock()
return multierr.Combine(t.active.DeleteMany(ks), t.inactive.DeleteMany(ks))
}
func (t *TimedCacheBlockstore) AllKeysChan(_ context.Context) (<-chan cid.Cid, error) {
t.mu.RLock()
defer t.mu.RUnlock()

9
blockstore/union.go
View File

@ -82,6 +82,15 @@ func (m unionBlockstore) DeleteBlock(cid cid.Cid) (err error) {
return err
}
func (m unionBlockstore) DeleteMany(cids []cid.Cid) (err error) {
for _, bs := range m {
if err = bs.DeleteMany(cids); err != nil {
break
}
}
return err
}
func (m unionBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error) {
// this does not deduplicate; this interface needs to be revisited.
outCh := make(chan cid.Cid)

4
build/bootstrap/butterflynet.pi
View File

@ -1,2 +1,2 @@
/dns4/bootstrap-0.butterfly.fildev.network/tcp/1347/p2p/12D3KooWBQb5Eg2DRqL1Xrzm8S7AFnG2gkj1iKQKBBMEXJ7mXuWQ
/dns4/bootstrap-1.butterfly.fildev.network/tcp/1347/p2p/12D3KooWJ2qKfGJg2i4Pn7nVCQ13oktS4eZfXFJk9ZjQy9uxLdq9
/dns4/bootstrap-0.butterfly.fildev.network/tcp/1347/p2p/12D3KooWQafkXgEWDgcVhZvF6KMhiC8ktdxjvmdQN8RarRXe9jCc
/dns4/bootstrap-1.butterfly.fildev.network/tcp/1347/p2p/12D3KooWE7UmZ4DLk9WBdEJUSwuSCPiSqjoCv3wPeoe8Tq3yMa77

2
build/bootstrap/mainnet.pi
View File

@ -7,7 +7,7 @@
/dns4/bootstrap-6.mainnet.filops.net/tcp/1347/p2p/12D3KooWP5MwCiqdMETF9ub1P3MbCvQCcfconnYHbWg6sUJcDRQQ
/dns4/bootstrap-7.mainnet.filops.net/tcp/1347/p2p/12D3KooWRs3aY1p3juFjPy8gPN95PEQChm2QKGUCAdcDCC4EBMKf
/dns4/bootstrap-8.mainnet.filops.net/tcp/1347/p2p/12D3KooWScFR7385LTyR4zU1bYdzSiiAb5rnNABfVahPvVSzyTkR
/dns4/lotus-bootstrap.forceup.cn/tcp/41778/p2p/12D3KooWFQsv3nRMUevZNWWsY1Wu6NUzUbawnWU5NcRhgKuJA37C
/dns4/lotus-bootstrap.ipfsforce.com/tcp/41778/p2p/12D3KooWGhufNmZHF3sv48aQeS13ng5XVJZ9E6qy2Ms4VzqeUsHk
/dns4/bootstrap-0.starpool.in/tcp/12757/p2p/12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz
/dns4/bootstrap-1.starpool.in/tcp/12757/p2p/12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u
/dns4/node.glif.io/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt

8
build/bootstrap/nerpanet.pi
View File

@ -1,4 +1,4 @@
/dns4/bootstrap-0.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWNfuGjtzWTVz8eJGZ2C3aJg2xLqorhsagu4LTWw6CwpK9
/dns4/bootstrap-1.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWDfsxYk7dC6NNsHqZqqyMJCzkjZuXhjsmqBk3TUCBZLga
/dns4/bootstrap-2.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWRZAGHmCCaa2gkYmnC4Q2TEwHGFSh6Fh1FFJ7RSXak5yN
/dns4/bootstrap-3.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWBFxEigSKLvxJVdw3JziC9ePHHnyAn5LifWSqg2kttcth
/dns4/bootstrap-2.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWQcL6ReWmR6ASWx4iT7EiAmxKDQpvgq1MKNTQZp5NPnWW
/dns4/bootstrap-0.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWGyJCwCm7EfupM15CFPXM4c7zRVHwwwjcuy9umaGeztMX
/dns4/bootstrap-3.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWNK9RmfksKXSCQj7ZwAM7L6roqbN4kwJteihq7yPvSgPs
/dns4/bootstrap-1.nerpa.interplanetary.dev/tcp/1347/p2p/12D3KooWCWSaH6iUyXYspYxELjDfzToBsyVGVz3QvC7ysXv7wESo

BIN
build/genesis/butterflynet.car
View File

Binary file not shown.

BIN
build/genesis/nerpanet.car
View File

Binary file not shown.

43
build/openrpc.go Normal file
View File

@ -0,0 +1,43 @@
package build
import (
"bytes"
"compress/gzip"
"encoding/json"
rice "github.com/GeertJohan/go.rice"
apitypes "github.com/filecoin-project/lotus/api/types"
)
func mustReadGzippedOpenRPCDocument(data []byte) apitypes.OpenRPCDocument {
zr, err := gzip.NewReader(bytes.NewBuffer(data))
if err != nil {
log.Fatal(err)
}
m := apitypes.OpenRPCDocument{}
err = json.NewDecoder(zr).Decode(&m)
if err != nil {
log.Fatal(err)
}
err = zr.Close()
if err != nil {
log.Fatal(err)
}
return m
}
func OpenRPCDiscoverJSON_Full() apitypes.OpenRPCDocument {
data := rice.MustFindBox("openrpc").MustBytes("full.json.gz")
return mustReadGzippedOpenRPCDocument(data)
}
func OpenRPCDiscoverJSON_Miner() apitypes.OpenRPCDocument {
data := rice.MustFindBox("openrpc").MustBytes("miner.json.gz")
return mustReadGzippedOpenRPCDocument(data)
}
func OpenRPCDiscoverJSON_Worker() apitypes.OpenRPCDocument {
data := rice.MustFindBox("openrpc").MustBytes("worker.json.gz")
return mustReadGzippedOpenRPCDocument(data)
}

BIN
build/openrpc/full.json.gz Normal file
View File

Binary file not shown.

BIN
build/openrpc/miner.json.gz Normal file
View File

Binary file not shown.

BIN
build/openrpc/worker.json.gz Normal file
View File

Binary file not shown.

23
build/openrpc_test.go Normal file
View File

@ -0,0 +1,23 @@
package build
import (
"testing"
apitypes "github.com/filecoin-project/lotus/api/types"
)
func TestOpenRPCDiscoverJSON_Version(t *testing.T) {
// openRPCDocVersion is the current OpenRPC version of the API docs.
openRPCDocVersion := "1.2.6"
for i, docFn := range []func() apitypes.OpenRPCDocument{
OpenRPCDiscoverJSON_Full,
OpenRPCDiscoverJSON_Miner,
OpenRPCDiscoverJSON_Worker,
} {
doc := docFn()
if got, ok := doc["openrpc"]; !ok || got != openRPCDocVersion {
t.Fatalf("case: %d, want: %s, got: %v, doc: %v", i, openRPCDocVersion, got, doc)
}
}
}

18
build/params_nerpanet.go
View File

@ -24,21 +24,21 @@ const UpgradeSmokeHeight = -1
const UpgradeIgnitionHeight = -2
const UpgradeRefuelHeight = -3
const UpgradeTapeHeight = -4
const UpgradeLiftoffHeight = -5
const UpgradeActorsV2Height = 120 // critical: the network can bootstrap from v1 only
const UpgradeKumquatHeight = -6
const UpgradeActorsV2Height = 30 // critical: the network can bootstrap from v1 only
const UpgradeTapeHeight = 60
const UpgradeCalicoHeight = 306000
const UpgradePersianHeight = UpgradeCalicoHeight + (builtin2.EpochsInHour * 12)
const UpgradeKumquatHeight = 90
const UpgradeOrangeHeight = 307500
const UpgradeCalicoHeight = 100
const UpgradePersianHeight = UpgradeCalicoHeight + (builtin2.EpochsInHour * 1)
const UpgradeClausHeight = 307600
const UpgradeClausHeight = 250
const UpgradeActorsV3Height = 308000
const UpgradeOrangeHeight = 300
const UpgradeActorsV3Height = 600
func init() {
// Minimum block production power is set to 4 TiB

2
build/tools.go
View File

@ -3,7 +3,7 @@
package build
import (
_ "github.com/golang/mock/mockgen"
_ "github.com/GeertJohan/go.rice/rice"
_ "github.com/golang/mock/mockgen"
_ "github.com/whyrusleeping/bencher"
)

72
build/version.go
View File

@ -1,11 +1,5 @@
package build
import (
"fmt"
"golang.org/x/xerrors"
)
var CurrentCommit string
var BuildType int
@ -35,72 +29,8 @@ func buildType() string {
}
// BuildVersion is the local build version, set by build system
const BuildVersion = "1.5.0"
const BuildVersion = "1.6.0-dev"
func UserVersion() string {
return BuildVersion + buildType() + CurrentCommit
}
type Version uint32
func newVer(major, minor, patch uint8) Version {
return Version(uint32(major)<<16 | uint32(minor)<<8 | uint32(patch))
}
// Ints returns (major, minor, patch) versions
func (ve Version) Ints() (uint32, uint32, uint32) {
v := uint32(ve)
return (v & majorOnlyMask) >> 16, (v & minorOnlyMask) >> 8, v & patchOnlyMask
}
func (ve Version) String() string {
vmj, vmi, vp := ve.Ints()
return fmt.Sprintf("%d.%d.%d", vmj, vmi, vp)
}
func (ve Version) EqMajorMinor(v2 Version) bool {
return ve&minorMask == v2&minorMask
}
type NodeType int
const (
NodeUnknown NodeType = iota
NodeFull
NodeMiner
NodeWorker
)
var RunningNodeType NodeType
func VersionForType(nodeType NodeType) (Version, error) {
switch nodeType {
case NodeFull:
return FullAPIVersion, nil
case NodeMiner:
return MinerAPIVersion, nil
case NodeWorker:
return WorkerAPIVersion, nil
default:
return Version(0), xerrors.Errorf("unknown node type %d", nodeType)
}
}
// semver versions of the rpc api exposed
var (
FullAPIVersion = newVer(1, 1, 0)
MinerAPIVersion = newVer(1, 0, 1)
WorkerAPIVersion = newVer(1, 0, 0)
)
//nolint:varcheck,deadcode
const (
majorMask = 0xff0000
minorMask = 0xffff00
patchMask = 0xffffff
majorOnlyMask = 0xff0000
minorOnlyMask = 0x00ff00
patchOnlyMask = 0x0000ff
)

54
chain/events/events.go
View File

@ -55,11 +55,11 @@ type Events struct {
heightEvents
*hcEvents
observers []TipSetObserver
}
func NewEvents(ctx context.Context, api eventAPI) *Events {
gcConfidence := 2 * build.ForkLengthThreshold
func NewEventsWithConfidence(ctx context.Context, api eventAPI, gcConfidence abi.ChainEpoch) *Events {
tsc := newTSCache(gcConfidence, api)
e := &Events{
@ -77,8 +77,9 @@ func NewEvents(ctx context.Context, api eventAPI) *Events {
htHeights: map[abi.ChainEpoch][]uint64{},
},
hcEvents: newHCEvents(ctx, api, tsc, uint64(gcConfidence)),
ready: make(chan struct{}),
hcEvents: newHCEvents(ctx, api, tsc, uint64(gcConfidence)),
ready: make(chan struct{}),
observers: []TipSetObserver{},
}
go e.listenHeadChanges(ctx)
@ -92,6 +93,11 @@ func NewEvents(ctx context.Context, api eventAPI) *Events {
return e
}
func NewEvents(ctx context.Context, api eventAPI) *Events {
gcConfidence := 2 * build.ForkLengthThreshold
return NewEventsWithConfidence(ctx, api, gcConfidence)
}
func (e *Events) listenHeadChanges(ctx context.Context) {
for {
if err := e.listenHeadChangesOnce(ctx); err != nil {
@ -164,7 +170,7 @@ func (e *Events) listenHeadChangesOnce(ctx context.Context) error {
}
}
if err := e.headChange(rev, app); err != nil {
if err := e.headChange(ctx, rev, app); err != nil {
log.Warnf("headChange failed: %s", err)
}
@ -177,7 +183,7 @@ func (e *Events) listenHeadChangesOnce(ctx context.Context) error {
return nil
}
func (e *Events) headChange(rev, app []*types.TipSet) error {
func (e *Events) headChange(ctx context.Context, rev, app []*types.TipSet) error {
if len(app) == 0 {
return xerrors.New("events.headChange expected at least one applied tipset")
}
@ -189,5 +195,39 @@ func (e *Events) headChange(rev, app []*types.TipSet) error {
return err
}
if err := e.observeChanges(ctx, rev, app); err != nil {
return err
}
return e.processHeadChangeEvent(rev, app)
}
// A TipSetObserver receives notifications of tipsets
type TipSetObserver interface {
Apply(ctx context.Context, ts *types.TipSet) error
Revert(ctx context.Context, ts *types.TipSet) error
}
// TODO: add a confidence level so we can have observers with difference levels of confidence
func (e *Events) Observe(obs TipSetObserver) error {
e.lk.Lock()
defer e.lk.Unlock()
e.observers = append(e.observers, obs)
return nil
}
// observeChanges expects caller to hold e.lk
func (e *Events) observeChanges(ctx context.Context, rev, app []*types.TipSet) error {
for _, ts := range rev {
for _, o := range e.observers {
_ = o.Revert(ctx, ts)
}
}
for _, ts := range app {
for _, o := range e.observers {
_ = o.Apply(ctx, ts)
}
}
return nil
}

68
chain/gen/genesis/genesis.go
View File

@ -7,6 +7,7 @@ import (
"fmt"
"github.com/filecoin-project/lotus/chain/actors/builtin"
"github.com/filecoin-project/lotus/journal"
"github.com/ipfs/go-cid"
@ -233,13 +234,36 @@ func MakeInitialStateTree(ctx context.Context, bs bstore.Blockstore, template ge
}
vregroot, err := address.NewIDAddress(80)
if err != nil {
return nil, nil, err
}
switch template.VerifregRootKey.Type {
case genesis.TAccount:
var ainfo genesis.AccountMeta
if err := json.Unmarshal(template.VerifregRootKey.Meta, &ainfo); err != nil {
return nil, nil, xerrors.Errorf("unmarshaling account meta: %w", err)
}
st, err := cst.Put(ctx, &account0.State{Address: ainfo.Owner})
if err != nil {
return nil, nil, err
}
if err = createMultisigAccount(ctx, bs, cst, state, vregroot, template.VerifregRootKey, keyIDs); err != nil {
return nil, nil, xerrors.Errorf("failed to set up verified registry signer: %w", err)
_, ok := keyIDs[ainfo.Owner]
if ok {
return nil, nil, fmt.Errorf("rootkey account has already been declared, cannot be assigned 80: %s", ainfo.Owner)
}
err = state.SetActor(builtin.RootVerifierAddress, &types.Actor{
Code: builtin0.AccountActorCodeID,
Balance: template.VerifregRootKey.Balance,
Head: st,
})
if err != nil {
return nil, nil, xerrors.Errorf("setting verifreg rootkey account: %w", err)
}
case genesis.TMultisig:
if err = createMultisigAccount(ctx, bs, cst, state, builtin.RootVerifierAddress, template.VerifregRootKey, keyIDs); err != nil {
return nil, nil, xerrors.Errorf("failed to set up verified registry signer: %w", err)
}
default:
return nil, nil, xerrors.Errorf("unknown account type for verifreg rootkey: %w", err)
}
// Setup the first verifier as ID-address 81
@ -300,8 +324,36 @@ func MakeInitialStateTree(ctx context.Context, bs bstore.Blockstore, template ge
template.RemainderAccount.Balance = remainingFil
if err := createMultisigAccount(ctx, bs, cst, state, builtin.ReserveAddress, template.RemainderAccount, keyIDs); err != nil {
return nil, nil, xerrors.Errorf("failed to set up remainder account: %w", err)
switch template.RemainderAccount.Type {
case genesis.TAccount:
var ainfo genesis.AccountMeta
if err := json.Unmarshal(template.RemainderAccount.Meta, &ainfo); err != nil {
return nil, nil, xerrors.Errorf("unmarshaling account meta: %w", err)
}
st, err := cst.Put(ctx, &account0.State{Address: ainfo.Owner})
if err != nil {
return nil, nil, err
}
_, ok := keyIDs[ainfo.Owner]
if ok {
return nil, nil, fmt.Errorf("remainder account has already been declared, cannot be assigned 90: %s", ainfo.Owner)
}
err = state.SetActor(builtin.ReserveAddress, &types.Actor{
Code: builtin0.AccountActorCodeID,
Balance: template.RemainderAccount.Balance,
Head: st,
})
if err != nil {
return nil, nil, xerrors.Errorf("setting remainder account: %w", err)
}
case genesis.TMultisig:
if err = createMultisigAccount(ctx, bs, cst, state, builtin.ReserveAddress, template.RemainderAccount, keyIDs); err != nil {
return nil, nil, xerrors.Errorf("failed to set up remainder: %w", err)
}
default:
return nil, nil, xerrors.Errorf("unknown account type for remainder: %w", err)
}
return state, keyIDs, nil

4
chain/stmgr/forks.go
View File

@ -852,6 +852,10 @@ func UpgradeLiftoff(ctx context.Context, sm *StateManager, _ MigrationCache, cb
}
func UpgradeCalico(ctx context.Context, sm *StateManager, _ MigrationCache, cb ExecCallback, root cid.Cid, epoch abi.ChainEpoch, ts *types.TipSet) (cid.Cid, error) {
if build.BuildType != build.BuildMainnet {
return root, nil
}
store := sm.cs.ActorStore(ctx)
var stateRoot types.StateRoot
if err := store.Get(ctx, root, &stateRoot); err != nil {

2
node/modules/rpcstatemanager.go → chain/stmgr/rpc/rpcstatemanager.go
View File

@ -1,4 +1,4 @@
package modules
package rpcstmgr
import (
"context"

2
chain/store/store.go
View File

@ -81,7 +81,7 @@ func init() {
}
// ReorgNotifee represents a callback that gets called upon reorgs.
type ReorgNotifee func(rev, app []*types.TipSet) error
type ReorgNotifee = func(rev, app []*types.TipSet) error
// Journal event types.
const (

2
chain/sub/incoming.go
View File

@ -101,7 +101,7 @@ func HandleIncomingBlocks(ctx context.Context, bsub *pubsub.Subscription, s *cha
[]tag.Mutator{tag.Insert(metrics.MinerID, blk.Header.Miner.String())},
metrics.BlockDelay.M(delay),
)
log.Warnf("Received block with large delay %d from miner %s", delay, blk.Header.Miner)
log.Warnw("received block with large delay from miner", "block", blk.Cid(), "delay", delay, "miner", blk.Header.Miner)
}
if s.InformNewBlock(msg.ReceivedFrom, &types.FullBlock{

3
cli/auth.go
View File

@ -9,6 +9,7 @@ import (
"github.com/filecoin-project/go-jsonrpc/auth"
"github.com/filecoin-project/lotus/api/apistruct"
cliutil "github.com/filecoin-project/lotus/cli/util"
"github.com/filecoin-project/lotus/node/repo"
)
@ -127,7 +128,7 @@ var authApiInfoToken = &cli.Command{
// TODO: Log in audit log when it is implemented
fmt.Printf("%s=%s:%s\n", envForRepo(t), string(token), ainfo.Addr)
fmt.Printf("%s=%s:%s\n", cliutil.EnvForRepo(t), string(token), ainfo.Addr)
return nil
},
}

5
cli/backup.go
View File

@ -51,7 +51,10 @@ func BackupCmd(repoFlag string, rt repo.RepoType, getApi BackupApiFn) *cli.Comma
return xerrors.Errorf("getting metadata datastore: %w", err)
}
bds := backupds.Wrap(mds)
bds, err := backupds.Wrap(mds, backupds.NoLogdir)
if err != nil {
return err
}
fpath, err := homedir.Expand(cctx.Args().First())
if err != nil {

270
cli/cmd.go
View File

@ -1,34 +1,17 @@
package cli
import (
"context"
"fmt"
"net/http"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
logging "github.com/ipfs/go-log/v2"
"github.com/mitchellh/go-homedir"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-jsonrpc"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/client"
cliutil "github.com/filecoin-project/lotus/cli/util"
"github.com/filecoin-project/lotus/node/repo"
)
var log = logging.Logger("cli")
const (
metadataTraceContext = "traceContext"
)
// custom CLI error
type ErrCmdFailed struct {
@ -46,167 +29,6 @@ func NewCliError(s string) error {
// ApiConnector returns API instance
type ApiConnector func() api.FullNode
// The flag passed on the command line with the listen address of the API
// server (only used by the tests)
func flagForAPI(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "api-url"
case repo.StorageMiner:
return "miner-api-url"
case repo.Worker:
return "worker-api-url"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func flagForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "repo"
case repo.StorageMiner:
return "miner-repo"
case repo.Worker:
return "worker-repo"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func envForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO"
case repo.StorageMiner:
return "MINER_API_INFO"
case repo.Worker:
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
// TODO remove after deprecation period
func envForRepoDeprecation(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO"
case repo.StorageMiner:
return "STORAGE_API_INFO"
case repo.Worker:
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func GetAPIInfo(ctx *cli.Context, t repo.RepoType) (cliutil.APIInfo, error) {
// Check if there was a flag passed with the listen address of the API
// server (only used by the tests)
apiFlag := flagForAPI(t)
if ctx.IsSet(apiFlag) {
strma := ctx.String(apiFlag)
strma = strings.TrimSpace(strma)
return cliutil.APIInfo{Addr: strma}, nil
}
envKey := envForRepo(t)
env, ok := os.LookupEnv(envKey)
if !ok {
// TODO remove after deprecation period
envKey = envForRepoDeprecation(t)
env, ok = os.LookupEnv(envKey)
if ok {
log.Warnf("Use deprecation env(%s) value, please use env(%s) instead.", envKey, envForRepo(t))
}
}
if ok {
return cliutil.ParseApiInfo(env), nil
}
repoFlag := flagForRepo(t)
p, err := homedir.Expand(ctx.String(repoFlag))
if err != nil {
return cliutil.APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", repoFlag, err)
}
r, err := repo.NewFS(p)
if err != nil {
return cliutil.APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
}
ma, err := r.APIEndpoint()
if err != nil {
return cliutil.APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
}
token, err := r.APIToken()
if err != nil {
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
}
return cliutil.APIInfo{
Addr: ma.String(),
Token: token,
}, nil
}
func GetRawAPI(ctx *cli.Context, t repo.RepoType) (string, http.Header, error) {
ainfo, err := GetAPIInfo(ctx, t)
if err != nil {
return "", nil, xerrors.Errorf("could not get API info: %w", err)
}
addr, err := ainfo.DialArgs()
if err != nil {
return "", nil, xerrors.Errorf("could not get DialArgs: %w", err)
}
return addr, ainfo.AuthHeader(), nil
}
func GetAPI(ctx *cli.Context) (api.Common, jsonrpc.ClientCloser, error) {
ti, ok := ctx.App.Metadata["repoType"]
if !ok {
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
ti = repo.FullNode
}
t, ok := ti.(repo.RepoType)
if !ok {
log.Errorf("repoType type does not match the type of repo.RepoType")
}
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
return tn.(api.StorageMiner), func() {}, nil
}
if tn, ok := ctx.App.Metadata["testnode-full"]; ok {
return tn.(api.FullNode), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, t)
if err != nil {
return nil, nil, err
}
return client.NewCommonRPC(ctx.Context, addr, headers)
}
func GetFullNodeAPI(ctx *cli.Context) (api.FullNode, jsonrpc.ClientCloser, error) {
if tn, ok := ctx.App.Metadata["testnode-full"]; ok {
return tn.(api.FullNode), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, repo.FullNode)
if err != nil {
return nil, nil, err
}
return client.NewFullNodeRPC(ctx.Context, addr, headers)
}
func GetFullNodeServices(ctx *cli.Context) (ServicesAPI, error) {
if tn, ok := ctx.App.Metadata["test-services"]; ok {
return tn.(ServicesAPI), nil
@ -220,92 +42,18 @@ func GetFullNodeServices(ctx *cli.Context) (ServicesAPI, error) {
return &ServicesImpl{api: api, closer: c}, nil
}
type GetStorageMinerOptions struct {
PreferHttp bool
}
var GetAPIInfo = cliutil.GetAPIInfo
var GetRawAPI = cliutil.GetRawAPI
var GetAPI = cliutil.GetAPI
type GetStorageMinerOption func(*GetStorageMinerOptions)
var DaemonContext = cliutil.DaemonContext
var ReqContext = cliutil.ReqContext
func StorageMinerUseHttp(opts *GetStorageMinerOptions) {
opts.PreferHttp = true
}
var GetFullNodeAPI = cliutil.GetFullNodeAPI
var GetGatewayAPI = cliutil.GetGatewayAPI
func GetStorageMinerAPI(ctx *cli.Context, opts ...GetStorageMinerOption) (api.StorageMiner, jsonrpc.ClientCloser, error) {
var options GetStorageMinerOptions
for _, opt := range opts {
opt(&options)
}
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
return tn.(api.StorageMiner), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, repo.StorageMiner)
if err != nil {
return nil, nil, err
}
if options.PreferHttp {
u, err := url.Parse(addr)
if err != nil {
return nil, nil, xerrors.Errorf("parsing miner api URL: %w", err)
}
switch u.Scheme {
case "ws":
u.Scheme = "http"
case "wss":
u.Scheme = "https"
}
addr = u.String()
}
return client.NewStorageMinerRPC(ctx.Context, addr, headers)
}
func GetWorkerAPI(ctx *cli.Context) (api.WorkerAPI, jsonrpc.ClientCloser, error) {
addr, headers, err := GetRawAPI(ctx, repo.Worker)
if err != nil {
return nil, nil, err
}
return client.NewWorkerRPC(ctx.Context, addr, headers)
}
func GetGatewayAPI(ctx *cli.Context) (api.GatewayAPI, jsonrpc.ClientCloser, error) {
addr, headers, err := GetRawAPI(ctx, repo.FullNode)
if err != nil {
return nil, nil, err
}
return client.NewGatewayRPC(ctx.Context, addr, headers)
}
func DaemonContext(cctx *cli.Context) context.Context {
if mtCtx, ok := cctx.App.Metadata[metadataTraceContext]; ok {
return mtCtx.(context.Context)
}
return context.Background()
}
// ReqContext returns context for cli execution. Calling it for the first time
// installs SIGTERM handler that will close returned context.
// Not safe for concurrent execution.
func ReqContext(cctx *cli.Context) context.Context {
tCtx := DaemonContext(cctx)
ctx, done := context.WithCancel(tCtx)
sigChan := make(chan os.Signal, 2)
go func() {
<-sigChan
done()
}()
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
return ctx
}
var GetStorageMinerAPI = cliutil.GetStorageMinerAPI
var GetWorkerAPI = cliutil.GetWorkerAPI
var CommonCommands = []*cli.Command{
netCmd,

2
cli/disputer.go
View File

@ -270,7 +270,7 @@ var disputerStartCmd = &cli.Command{
if err != nil {
disputeLog.Errorw("failed to dispute post message", "err", err.Error(), "miner", dpmsg.To)
} else {
disputeLog.Infof("submited dispute", "mcid", m.Cid(), "miner", dpmsg.To)
disputeLog.Infow("submited dispute", "mcid", m.Cid(), "miner", dpmsg.To)
}
}

4
cli/multisig.go
View File

@ -1275,7 +1275,7 @@ var msigLockApproveCmd = &cli.Command{
params, actErr := actors.SerializeParams(&msig2.LockBalanceParams{
StartEpoch: abi.ChainEpoch(start),
UnlockDuration: abi.ChainEpoch(duration),
Amount: abi.NewTokenAmount(amount.Int64()),
Amount: big.Int(amount),
})
if actErr != nil {
@ -1367,7 +1367,7 @@ var msigLockCancelCmd = &cli.Command{
params, actErr := actors.SerializeParams(&msig2.LockBalanceParams{
StartEpoch: abi.ChainEpoch(start),
UnlockDuration: abi.ChainEpoch(duration),
Amount: abi.NewTokenAmount(amount.Int64()),
Amount: big.Int(amount),
})
if actErr != nil {

50
cli/net.go
View File

@ -48,6 +48,11 @@ var NetPeers = &cli.Command{
Aliases: []string{"a"},
Usage: "Print agent name",
},
&cli.BoolFlag{
Name: "extended",
Aliases: []string{"x"},
Usage: "Print extended peer information in json",
},
},
Action: func(cctx *cli.Context) error {
api, closer, err := GetAPI(cctx)
@ -65,18 +70,42 @@ var NetPeers = &cli.Command{
return strings.Compare(string(peers[i].ID), string(peers[j].ID)) > 0
})
for _, peer := range peers {
var agent string
if cctx.Bool("agent") {
agent, err = api.NetAgentVersion(ctx, peer.ID)
if cctx.Bool("extended") {
// deduplicate
seen := make(map[peer.ID]struct{})
for _, peer := range peers {
_, dup := seen[peer.ID]
if dup {
continue
}
seen[peer.ID] = struct{}{}
info, err := api.NetPeerInfo(ctx, peer.ID)
if err != nil {
log.Warnf("getting agent version: %s", err)
log.Warnf("error getting extended peer info: %s", err)
} else {
agent = ", " + agent
bytes, err := json.Marshal(&info)
if err != nil {
log.Warnf("error marshalling extended peer info: %s", err)
} else {
fmt.Println(string(bytes))
}
}
}
fmt.Printf("%s, %s%s\n", peer.ID, peer.Addrs, agent)
} else {
for _, peer := range peers {
var agent string
if cctx.Bool("agent") {
agent, err = api.NetAgentVersion(ctx, peer.ID)
if err != nil {
log.Warnf("getting agent version: %s", err)
} else {
agent = ", " + agent
}
}
fmt.Printf("%s, %s%s\n", peer.ID, peer.Addrs, agent)
}
}
return nil
@ -88,8 +117,9 @@ var netScores = &cli.Command{
Usage: "Print peers' pubsub scores",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "extended",
Usage: "print extended peer scores in json",
Name: "extended",
Aliases: []string{"x"},
Usage: "print extended peer scores in json",
},
},
Action: func(cctx *cli.Context) error {

274
cli/util/api.go Normal file
View File

@ -0,0 +1,274 @@
package cliutil
import (
"context"
"fmt"
"net/http"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"github.com/mitchellh/go-homedir"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-jsonrpc"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/client"
"github.com/filecoin-project/lotus/node/repo"
)
const (
metadataTraceContext = "traceContext"
)
// The flag passed on the command line with the listen address of the API
// server (only used by the tests)
func flagForAPI(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "api-url"
case repo.StorageMiner:
return "miner-api-url"
case repo.Worker:
return "worker-api-url"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func flagForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "repo"
case repo.StorageMiner:
return "miner-repo"
case repo.Worker:
return "worker-repo"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func EnvForRepo(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO"
case repo.StorageMiner:
return "MINER_API_INFO"
case repo.Worker:
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
// TODO remove after deprecation period
func envForRepoDeprecation(t repo.RepoType) string {
switch t {
case repo.FullNode:
return "FULLNODE_API_INFO"
case repo.StorageMiner:
return "STORAGE_API_INFO"
case repo.Worker:
return "WORKER_API_INFO"
default:
panic(fmt.Sprintf("Unknown repo type: %v", t))
}
}
func GetAPIInfo(ctx *cli.Context, t repo.RepoType) (APIInfo, error) {
// Check if there was a flag passed with the listen address of the API
// server (only used by the tests)
apiFlag := flagForAPI(t)
if ctx.IsSet(apiFlag) {
strma := ctx.String(apiFlag)
strma = strings.TrimSpace(strma)
return APIInfo{Addr: strma}, nil
}
envKey := EnvForRepo(t)
env, ok := os.LookupEnv(envKey)
if !ok {
// TODO remove after deprecation period
envKey = envForRepoDeprecation(t)
env, ok = os.LookupEnv(envKey)
if ok {
log.Warnf("Use deprecation env(%s) value, please use env(%s) instead.", envKey, EnvForRepo(t))
}
}
if ok {
return ParseApiInfo(env), nil
}
repoFlag := flagForRepo(t)
p, err := homedir.Expand(ctx.String(repoFlag))
if err != nil {
return APIInfo{}, xerrors.Errorf("could not expand home dir (%s): %w", repoFlag, err)
}
r, err := repo.NewFS(p)
if err != nil {
return APIInfo{}, xerrors.Errorf("could not open repo at path: %s; %w", p, err)
}
ma, err := r.APIEndpoint()
if err != nil {
return APIInfo{}, xerrors.Errorf("could not get api endpoint: %w", err)
}
token, err := r.APIToken()
if err != nil {
log.Warnf("Couldn't load CLI token, capabilities may be limited: %v", err)
}
return APIInfo{
Addr: ma.String(),
Token: token,
}, nil
}
func GetRawAPI(ctx *cli.Context, t repo.RepoType) (string, http.Header, error) {
ainfo, err := GetAPIInfo(ctx, t)
if err != nil {
return "", nil, xerrors.Errorf("could not get API info: %w", err)
}
addr, err := ainfo.DialArgs()
if err != nil {
return "", nil, xerrors.Errorf("could not get DialArgs: %w", err)
}
return addr, ainfo.AuthHeader(), nil
}
func GetAPI(ctx *cli.Context) (api.Common, jsonrpc.ClientCloser, error) {
ti, ok := ctx.App.Metadata["repoType"]
if !ok {
log.Errorf("unknown repo type, are you sure you want to use GetAPI?")
ti = repo.FullNode
}
t, ok := ti.(repo.RepoType)
if !ok {
log.Errorf("repoType type does not match the type of repo.RepoType")
}
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
return tn.(api.StorageMiner), func() {}, nil
}
if tn, ok := ctx.App.Metadata["testnode-full"]; ok {
return tn.(api.FullNode), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, t)
if err != nil {
return nil, nil, err
}
return client.NewCommonRPC(ctx.Context, addr, headers)
}
func GetFullNodeAPI(ctx *cli.Context) (api.FullNode, jsonrpc.ClientCloser, error) {
if tn, ok := ctx.App.Metadata["testnode-full"]; ok {
return tn.(api.FullNode), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, repo.FullNode)
if err != nil {
return nil, nil, err
}
return client.NewFullNodeRPC(ctx.Context, addr, headers)
}
type GetStorageMinerOptions struct {
PreferHttp bool
}
type GetStorageMinerOption func(*GetStorageMinerOptions)
func StorageMinerUseHttp(opts *GetStorageMinerOptions) {
opts.PreferHttp = true
}
func GetStorageMinerAPI(ctx *cli.Context, opts ...GetStorageMinerOption) (api.StorageMiner, jsonrpc.ClientCloser, error) {
var options GetStorageMinerOptions
for _, opt := range opts {
opt(&options)
}
if tn, ok := ctx.App.Metadata["testnode-storage"]; ok {
return tn.(api.StorageMiner), func() {}, nil
}
addr, headers, err := GetRawAPI(ctx, repo.StorageMiner)
if err != nil {
return nil, nil, err
}
if options.PreferHttp {
u, err := url.Parse(addr)
if err != nil {
return nil, nil, xerrors.Errorf("parsing miner api URL: %w", err)
}
switch u.Scheme {
case "ws":
u.Scheme = "http"
case "wss":
u.Scheme = "https"
}
addr = u.String()
}
return client.NewStorageMinerRPC(ctx.Context, addr, headers)
}
func GetWorkerAPI(ctx *cli.Context) (api.WorkerAPI, jsonrpc.ClientCloser, error) {
addr, headers, err := GetRawAPI(ctx, repo.Worker)
if err != nil {
return nil, nil, err
}
return client.NewWorkerRPC(ctx.Context, addr, headers)
}
func GetGatewayAPI(ctx *cli.Context) (api.GatewayAPI, jsonrpc.ClientCloser, error) {
addr, headers, err := GetRawAPI(ctx, repo.FullNode)
if err != nil {
return nil, nil, err
}
return client.NewGatewayRPC(ctx.Context, addr, headers)
}
func DaemonContext(cctx *cli.Context) context.Context {
if mtCtx, ok := cctx.App.Metadata[metadataTraceContext]; ok {
return mtCtx.(context.Context)
}
return context.Background()
}
// ReqContext returns context for cli execution. Calling it for the first time
// installs SIGTERM handler that will close returned context.
// Not safe for concurrent execution.
func ReqContext(cctx *cli.Context) context.Context {
tCtx := DaemonContext(cctx)
ctx, done := context.WithCancel(tCtx)
sigChan := make(chan os.Signal, 2)
go func() {
<-sigChan
done()
}()
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
return ctx
}

4
cmd/lotus-gateway/api.go
View File

@ -34,7 +34,7 @@ var (
// gatewayDepsAPI defines the API methods that the GatewayAPI depends on
// (to make it easy to mock for tests)
type gatewayDepsAPI interface {
Version(context.Context) (api.Version, error)
Version(context.Context) (api.APIVersion, error)
ChainGetBlockMessages(context.Context, cid.Cid) (*api.BlockMessages, error)
ChainGetMessage(ctx context.Context, mc cid.Cid) (*types.Message, error)
ChainGetNode(ctx context.Context, p string) (*api.IpldObject, error)
@ -130,7 +130,7 @@ func (a *GatewayAPI) checkTimestamp(at time.Time) error {
return nil
}
func (a *GatewayAPI) Version(ctx context.Context) (api.Version, error) {
func (a *GatewayAPI) Version(ctx context.Context) (api.APIVersion, error) {
return a.api.Version(ctx)
}

2
cmd/lotus-gateway/main.go
View File

@ -77,7 +77,7 @@ var runCmd = &cli.Command{
// Register all metric views
if err := view.Register(
metrics.DefaultViews...,
metrics.ChainNodeViews...,
); err != nil {
log.Fatalf("Cannot register the view: %v", err)
}

9
cmd/lotus-seal-worker/main.go
View File

@ -31,6 +31,7 @@ import (
"github.com/filecoin-project/lotus/api/apistruct"
"github.com/filecoin-project/lotus/build"
lcli "github.com/filecoin-project/lotus/cli"
cliutil "github.com/filecoin-project/lotus/cli/util"
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
"github.com/filecoin-project/lotus/extern/sector-storage/sealtasks"
"github.com/filecoin-project/lotus/extern/sector-storage/stores"
@ -49,7 +50,7 @@ const FlagWorkerRepo = "worker-repo"
const FlagWorkerRepoDeprecation = "workerrepo"
func main() {
build.RunningNodeType = build.NodeWorker
api.RunningNodeType = api.NodeWorker
lotuslog.SetupLogLevels()
@ -183,7 +184,7 @@ var runCmd = &cli.Command{
var closer func()
var err error
for {
nodeApi, closer, err = lcli.GetStorageMinerAPI(cctx, lcli.StorageMinerUseHttp)
nodeApi, closer, err = lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
if err == nil {
_, err = nodeApi.Version(ctx)
if err == nil {
@ -210,8 +211,8 @@ var runCmd = &cli.Command{
if err != nil {
return err
}
if v.APIVersion != build.MinerAPIVersion {
return xerrors.Errorf("lotus-miner API version doesn't match: expected: %s", api.Version{APIVersion: build.MinerAPIVersion})
if v.APIVersion != api.MinerAPIVersion {
return xerrors.Errorf("lotus-miner API version doesn't match: expected: %s", api.APIVersion{APIVersion: api.MinerAPIVersion})
}
log.Infof("Remote version %s", v)

10
cmd/lotus-seal-worker/rpc.go
View File

@ -8,6 +8,8 @@ import (
"github.com/mitchellh/go-homedir"
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/api"
apitypes "github.com/filecoin-project/lotus/api/types"
"github.com/filecoin-project/lotus/build"
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
"github.com/filecoin-project/lotus/extern/sector-storage/stores"
@ -23,8 +25,8 @@ type worker struct {
disabled int64
}
func (w *worker) Version(context.Context) (build.Version, error) {
return build.WorkerAPIVersion, nil
func (w *worker) Version(context.Context) (api.Version, error) {
return api.WorkerAPIVersion, nil
}
func (w *worker) StorageAddLocal(ctx context.Context, path string) error {
@ -76,4 +78,8 @@ func (w *worker) Session(ctx context.Context) (uuid.UUID, error) {
return w.LocalWorker.Session(ctx)
}
func (w *worker) Discover(ctx context.Context) (apitypes.OpenRPCDocument, error) {
return build.OpenRPCDiscoverJSON_Worker(), nil
}
var _ storiface.WorkerCalls = &worker{}

22
cmd/lotus-seal-worker/storage.go
View File

@ -6,6 +6,7 @@ import (
"os"
"path/filepath"
"github.com/docker/go-units"
"github.com/google/uuid"
"github.com/mitchellh/go-homedir"
"github.com/urfave/cli/v2"
@ -46,6 +47,10 @@ var storageAttachCmd = &cli.Command{
Name: "store",
Usage: "(for init) use path for long-term storage",
},
&cli.StringFlag{
Name: "max-storage",
Usage: "(for init) limit storage space for sectors (expensive for very large paths!)",
},
},
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetWorkerAPI(cctx)
@ -79,11 +84,20 @@ var storageAttachCmd = &cli.Command{
return err
}
var maxStor int64
if cctx.IsSet("max-storage") {
maxStor, err = units.RAMInBytes(cctx.String("max-storage"))
if err != nil {
return xerrors.Errorf("parsing max-storage: %w", err)
}
}
cfg := &stores.LocalStorageMeta{
ID: stores.ID(uuid.New().String()),
Weight: cctx.Uint64("weight"),
CanSeal: cctx.Bool("seal"),
CanStore: cctx.Bool("store"),
ID: stores.ID(uuid.New().String()),
Weight: cctx.Uint64("weight"),
CanSeal: cctx.Bool("seal"),
CanStore: cctx.Bool("store"),
MaxStorage: uint64(maxStor),
}
if !(cfg.CanStore || cfg.CanSeal) {

194
cmd/lotus-seed/genesis.go
View File

@ -37,6 +37,8 @@ var genesisCmd = &cli.Command{
genesisNewCmd,
genesisAddMinerCmd,
genesisAddMsigsCmd,
genesisSetVRKCmd,
genesisSetRemainderCmd,
genesisCarCmd,
},
}
@ -309,6 +311,198 @@ func parseMultisigCsv(csvf string) ([]GenAccountEntry, error) {
return entries, nil
}
var genesisSetVRKCmd = &cli.Command{
Name: "set-vrk",
Usage: "Set the verified registry's root key",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "multisig",
Usage: "CSV file to parse the multisig that will be set as the root key",
},
&cli.StringFlag{
Name: "account",
Usage: "pubkey address that will be set as the root key (must NOT be declared anywhere else, since it must be given ID 80)",
},
},
Action: func(cctx *cli.Context) error {
if cctx.Args().Len() != 1 {
return fmt.Errorf("must specify template file")
}
genf, err := homedir.Expand(cctx.Args().First())
if err != nil {
return err
}
var template genesis.Template
b, err := ioutil.ReadFile(genf)
if err != nil {
return xerrors.Errorf("read genesis template: %w", err)
}
if err := json.Unmarshal(b, &template); err != nil {
return xerrors.Errorf("unmarshal genesis template: %w", err)
}
if cctx.IsSet("account") {
addr, err := address.NewFromString(cctx.String("account"))
if err != nil {
return err
}
am := genesis.AccountMeta{Owner: addr}
template.VerifregRootKey = genesis.Actor{
Type: genesis.TAccount,
Balance: big.Zero(),
Meta: am.ActorMeta(),
}
} else if cctx.IsSet("multisig") {
csvf, err := homedir.Expand(cctx.String("multisig"))
if err != nil {
return err
}
entries, err := parseMultisigCsv(csvf)
if err != nil {
return xerrors.Errorf("parsing multisig csv file: %w", err)
}
if len(entries) == 0 {
return xerrors.Errorf("no msig entries in csv file: %w", err)
}
e := entries[0]
if len(e.Addresses) != e.N {
return fmt.Errorf("entry had mismatch between 'N' and number of addresses")
}
msig := &genesis.MultisigMeta{
Signers: e.Addresses,
Threshold: e.M,
VestingDuration: monthsToBlocks(e.VestingMonths),
VestingStart: 0,
}
act := genesis.Actor{
Type: genesis.TMultisig,
Balance: abi.TokenAmount(e.Amount),
Meta: msig.ActorMeta(),
}
template.VerifregRootKey = act
} else {
return xerrors.Errorf("must include either --account or --multisig flag")
}
b, err = json.MarshalIndent(&template, "", " ")
if err != nil {
return err
}
if err := ioutil.WriteFile(genf, b, 0644); err != nil {
return err
}
return nil
},
}
var genesisSetRemainderCmd = &cli.Command{
Name: "set-remainder",
Usage: "Set the remainder actor",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "multisig",
Usage: "CSV file to parse the multisig that will be set as the remainder actor",
},
&cli.StringFlag{
Name: "account",
Usage: "pubkey address that will be set as the remainder key (must NOT be declared anywhere else, since it must be given ID 90)",
},
},
Action: func(cctx *cli.Context) error {
if cctx.Args().Len() != 1 {
return fmt.Errorf("must specify template file")
}
genf, err := homedir.Expand(cctx.Args().First())
if err != nil {
return err
}
var template genesis.Template
b, err := ioutil.ReadFile(genf)
if err != nil {
return xerrors.Errorf("read genesis template: %w", err)
}
if err := json.Unmarshal(b, &template); err != nil {
return xerrors.Errorf("unmarshal genesis template: %w", err)
}
if cctx.IsSet("account") {
addr, err := address.NewFromString(cctx.String("account"))
if err != nil {
return err
}
am := genesis.AccountMeta{Owner: addr}
template.RemainderAccount = genesis.Actor{
Type: genesis.TAccount,
Balance: big.Zero(),
Meta: am.ActorMeta(),
}
} else if cctx.IsSet("multisig") {
csvf, err := homedir.Expand(cctx.String("multisig"))
if err != nil {
return err
}
entries, err := parseMultisigCsv(csvf)
if err != nil {
return xerrors.Errorf("parsing multisig csv file: %w", err)
}
if len(entries) == 0 {
return xerrors.Errorf("no msig entries in csv file: %w", err)
}
e := entries[0]
if len(e.Addresses) != e.N {
return fmt.Errorf("entry had mismatch between 'N' and number of addresses")
}
msig := &genesis.MultisigMeta{
Signers: e.Addresses,
Threshold: e.M,
VestingDuration: monthsToBlocks(e.VestingMonths),
VestingStart: 0,
}
act := genesis.Actor{
Type: genesis.TMultisig,
Balance: abi.TokenAmount(e.Amount),
Meta: msig.ActorMeta(),
}
template.RemainderAccount = act
} else {
return xerrors.Errorf("must include either --account or --multisig flag")
}
b, err = json.MarshalIndent(&template, "", " ")
if err != nil {
return err
}
if err := ioutil.WriteFile(genf, b, 0644); err != nil {
return err
}
return nil
},
}
var genesisCarCmd = &cli.Command{
Name: "car",
Description: "write genesis car file",

2
cmd/lotus-shed/consensus.go
View File

@ -36,7 +36,7 @@ type consensusItem struct {
targetTipset *types.TipSet
headTipset *types.TipSet
peerID peer.ID
version api.Version
version api.APIVersion
api api.FullNode
}

3
cmd/lotus-shed/main.go
View File

@ -37,10 +37,11 @@ func main() {
mpoolCmd,
genesisVerifyCmd,
mathCmd,
minerCmd,
mpoolStatsCmd,
exportChainCmd,
consensusCmd,
rollupDealStatsCmd,
storageStatsCmd,
syncCmd,
stateTreePruneCmd,
datastoreCmd,

113
cmd/lotus-shed/miner.go Normal file
View File

@ -0,0 +1,113 @@
package main
import (
"bufio"
"io"
"os"
"path/filepath"
"strings"
"github.com/mitchellh/go-homedir"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
)
var minerCmd = &cli.Command{
Name: "miner",
Usage: "miner-related utilities",
Subcommands: []*cli.Command{
minerUnpackInfoCmd,
},
}
var minerUnpackInfoCmd = &cli.Command{
Name: "unpack-info",
Usage: "unpack miner info all dump",
ArgsUsage: "[allinfo.txt] [dir]",
Action: func(cctx *cli.Context) error {
if cctx.Args().Len() != 2 {
return xerrors.Errorf("expected 2 args")
}
src, err := homedir.Expand(cctx.Args().Get(0))
if err != nil {
return xerrors.Errorf("expand src: %w", err)
}
f, err := os.Open(src)
if err != nil {
return xerrors.Errorf("open file: %w", err)
}
defer f.Close() // nolint
dest, err := homedir.Expand(cctx.Args().Get(1))
if err != nil {
return xerrors.Errorf("expand dest: %w", err)
}
var outf *os.File
r := bufio.NewReader(f)
for {
l, _, err := r.ReadLine()
if err == io.EOF {
if outf != nil {
return outf.Close()
}
}
if err != nil {
return xerrors.Errorf("read line: %w", err)
}
sl := string(l)
if strings.HasPrefix(sl, "#") {
if strings.Contains(sl, "..") {
return xerrors.Errorf("bad name %s", sl)
}
if strings.HasPrefix(sl, "#: ") {
if outf != nil {
if err := outf.Close(); err != nil {
return xerrors.Errorf("close out file: %w", err)
}
}
p := filepath.Join(dest, sl[len("#: "):])
if err := os.MkdirAll(filepath.Dir(p), 0775); err != nil {
return xerrors.Errorf("mkdir: %w", err)
}
outf, err = os.Create(p)
if err != nil {
return xerrors.Errorf("create out file: %w", err)
}
continue
}
if strings.HasPrefix(sl, "##: ") {
if outf != nil {
if err := outf.Close(); err != nil {
return xerrors.Errorf("close out file: %w", err)
}
}
p := filepath.Join(dest, "Per Sector Infos", sl[len("##: "):])
if err := os.MkdirAll(filepath.Dir(p), 0775); err != nil {
return xerrors.Errorf("mkdir: %w", err)
}
outf, err = os.Create(p)
if err != nil {
return xerrors.Errorf("create out file: %w", err)
}
continue
}
}
if outf != nil {
if _, err := outf.Write(l); err != nil {
return xerrors.Errorf("write line: %w", err)
}
if _, err := outf.Write([]byte("\n")); err != nil {
return xerrors.Errorf("write line end: %w", err)
}
}
}
},
}

455
cmd/lotus-shed/sr2-dealstats-rollup.go
View File

@ -1,455 +0,0 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"sort"
"strings"
"github.com/Jeffail/gabs"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/ipfs/go-cid"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
)
// Requested by @jbenet
// How many epochs back to look at for dealstats
var epochLookback = abi.ChainEpoch(10)
var resolvedWallets = map[address.Address]address.Address{}
var knownAddrMap = map[address.Address]string{}
//
// contents of basic_stats.json
type competitionTotalOutput struct {
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload competitionTotal `json:"payload"`
}
type competitionTotal struct {
UniqueCids int `json:"total_unique_cids"`
UniqueProviders int `json:"total_unique_providers"`
UniqueProjects int `json:"total_unique_projects"`
UniqueClients int `json:"total_unique_clients"`
TotalDeals int `json:"total_num_deals"`
TotalBytes int64 `json:"total_stored_data_size"`
FilplusTotalDeals int `json:"filplus_total_num_deals"`
FilplusTotalBytes int64 `json:"filplus_total_stored_data_size"`
seenProject map[string]bool
seenClient map[address.Address]bool
seenProvider map[address.Address]bool
seenPieceCid map[cid.Cid]bool
}
//
// contents of client_stats.json
type projectAggregateStatsOutput struct {
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload map[string]*projectAggregateStats `json:"payload"`
}
type projectAggregateStats struct {
ProjectID string `json:"project_id"`
DataSizeMaxProvider int64 `json:"max_data_size_stored_with_single_provider"`
HighestCidDealCount int `json:"max_same_cid_deals"`
DataSize int64 `json:"total_data_size"`
NumCids int `json:"total_num_cids"`
NumDeals int `json:"total_num_deals"`
NumProviders int `json:"total_num_providers"`
ClientStats map[string]*clientAggregateStats `json:"clients"`
dataPerProvider map[address.Address]int64
cidDeals map[cid.Cid]int
}
type clientAggregateStats struct {
Client string `json:"client"`
DataSize int64 `json:"total_data_size"`
NumCids int `json:"total_num_cids"`
NumDeals int `json:"total_num_deals"`
NumProviders int `json:"total_num_providers"`
providers map[address.Address]bool
cids map[cid.Cid]bool
}
//
// contents of deals_list_{{projid}}.json
type dealListOutput struct {
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload []*individualDeal `json:"payload"`
}
type individualDeal struct {
ProjectID string `json:"project_id"`
Client string `json:"client"`
DealID string `json:"deal_id"`
DealStartEpoch int64 `json:"deal_start_epoch"`
MinerID string `json:"miner_id"`
PayloadCID string `json:"payload_cid"`
PaddedSize int64 `json:"data_size"`
}
var rollupDealStatsCmd = &cli.Command{
Name: "rollup-deal-stats",
Flags: []cli.Flag{},
Action: func(cctx *cli.Context) error {
if cctx.Args().Len() != 2 || cctx.Args().Get(0) == "" || cctx.Args().Get(1) == "" {
return errors.New("must supply 2 arguments: a nonexistent target directory to write results to and a source of currently active projects")
}
outDirName := cctx.Args().Get(0)
if _, err := os.Stat(outDirName); err == nil {
return fmt.Errorf("unable to proceed: supplied stat target '%s' already exists", outDirName)
}
if err := os.MkdirAll(outDirName, 0755); err != nil {
return fmt.Errorf("creation of destination '%s' failed: %s", outDirName, err)
}
ctx := lcli.ReqContext(cctx)
projListName := cctx.Args().Get(1)
var projListFh *os.File
{
// Parses JSON input in the form:
// {
// "payload": [
// {
// "project": "5fb5f5b3ad3275e236287ce3",
// "address": "f3w3r2c6iukyh3u6f6kx62s5g6n2gf54aqp33ukqrqhje2y6xhf7k55przg4xqgahpcdal6laljz6zonma5pka"
// },
// {
// "project": "5fb608c4ad3275e236287ced",
// "address": "f3rs2khurnubol6ent27lpggidxxujqo2lg5aap5d5bmtam6yjb5wfla5cxxdgj45tqoaawgpzt5lofc3vpzfq"
// },
// ...
// ]
// }
if strings.HasPrefix(projListName, "http://") || strings.HasPrefix(projListName, "https://") {
req, err := http.NewRequestWithContext(ctx, "GET", projListName, nil)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close() //nolint:errcheck
if resp.StatusCode != http.StatusOK {
return xerrors.Errorf("non-200 response: %d", resp.StatusCode)
}
projListFh, err = os.Create(outDirName + "/client_list.json")
if err != nil {
return err
}
_, err = io.Copy(projListFh, resp.Body)
if err != nil {
return err
}
} else {
return errors.New("file inputs not yet supported")
}
if _, err := projListFh.Seek(0, 0); err != nil {
return err
}
defer projListFh.Close() //nolint:errcheck
projList, err := gabs.ParseJSONBuffer(projListFh)
if err != nil {
return err
}
proj, err := projList.Search("payload").Children()
if err != nil {
return err
}
for _, p := range proj {
a, err := address.NewFromString(p.S("address").Data().(string))
if err != nil {
return err
}
knownAddrMap[a] = p.S("project").Data().(string)
}
if len(knownAddrMap) == 0 {
return fmt.Errorf("no active projects/clients found in '%s': unable to continue", projListName)
}
}
outClientStatsFd, err := os.Create(outDirName + "/client_stats.json")
if err != nil {
return err
}
defer outClientStatsFd.Close() //nolint:errcheck
outBasicStatsFd, err := os.Create(outDirName + "/basic_stats.json")
if err != nil {
return err
}
defer outBasicStatsFd.Close() //nolint:errcheck
outUnfilteredStatsFd, err := os.Create(outDirName + "/unfiltered_basic_stats.json")
if err != nil {
return err
}
defer outUnfilteredStatsFd.Close() //nolint:errcheck
api, apiCloser, err := lcli.GetFullNodeAPI(cctx)
if err != nil {
return err
}
defer apiCloser()
head, err := api.ChainHead(ctx)
if err != nil {
return err
}
head, err = api.ChainGetTipSetByHeight(ctx, head.Height()-epochLookback, head.Key())
if err != nil {
return err
}
grandTotals := competitionTotal{
seenProject: make(map[string]bool),
seenClient: make(map[address.Address]bool),
seenProvider: make(map[address.Address]bool),
seenPieceCid: make(map[cid.Cid]bool),
}
unfilteredGrandTotals := competitionTotal{
seenClient: make(map[address.Address]bool),
seenProvider: make(map[address.Address]bool),
seenPieceCid: make(map[cid.Cid]bool),
}
projStats := make(map[string]*projectAggregateStats)
projDealLists := make(map[string][]*individualDeal)
deals, err := api.StateMarketDeals(ctx, head.Key())
if err != nil {
return err
}
for dealID, dealInfo := range deals {
// Only count deals that have properly started, not past/future ones
// https://github.com/filecoin-project/specs-actors/blob/v0.9.9/actors/builtin/market/deal.go#L81-L85
// Bail on 0 as well in case SectorStartEpoch is uninitialized due to some bug
if dealInfo.State.SectorStartEpoch <= 0 ||
dealInfo.State.SectorStartEpoch > head.Height() {
continue
}
clientAddr, found := resolvedWallets[dealInfo.Proposal.Client]
if !found {
var err error
clientAddr, err = api.StateAccountKey(ctx, dealInfo.Proposal.Client, head.Key())
if err != nil {
log.Warnf("failed to resolve id '%s' to wallet address: %s", dealInfo.Proposal.Client, err)
continue
}
resolvedWallets[dealInfo.Proposal.Client] = clientAddr
}
unfilteredGrandTotals.seenClient[clientAddr] = true
unfilteredGrandTotals.TotalBytes += int64(dealInfo.Proposal.PieceSize)
unfilteredGrandTotals.seenProvider[dealInfo.Proposal.Provider] = true
unfilteredGrandTotals.seenPieceCid[dealInfo.Proposal.PieceCID] = true
unfilteredGrandTotals.TotalDeals++
if dealInfo.Proposal.VerifiedDeal {
unfilteredGrandTotals.FilplusTotalDeals++
unfilteredGrandTotals.FilplusTotalBytes += int64(dealInfo.Proposal.PieceSize)
}
// perl -E 'say scalar gmtime ( 166560 * 30 + 1598306400 )'
// Wed Oct 21 18:00:00 2020
if dealInfo.Proposal.StartEpoch <= 166560 {
continue
}
projID, projKnown := knownAddrMap[clientAddr]
if !projKnown {
continue
}
grandTotals.seenProject[projID] = true
projStatEntry, ok := projStats[projID]
if !ok {
projStatEntry = &projectAggregateStats{
ProjectID: projID,
ClientStats: make(map[string]*clientAggregateStats),
cidDeals: make(map[cid.Cid]int),
dataPerProvider: make(map[address.Address]int64),
}
projStats[projID] = projStatEntry
}
if projStatEntry.cidDeals[dealInfo.Proposal.PieceCID] >= 10 {
continue
}
grandTotals.seenClient[clientAddr] = true
clientStatEntry, ok := projStatEntry.ClientStats[clientAddr.String()]
if !ok {
clientStatEntry = &clientAggregateStats{
Client: clientAddr.String(),
cids: make(map[cid.Cid]bool),
providers: make(map[address.Address]bool),
}
projStatEntry.ClientStats[clientAddr.String()] = clientStatEntry
}
grandTotals.TotalBytes += int64(dealInfo.Proposal.PieceSize)
projStatEntry.DataSize += int64(dealInfo.Proposal.PieceSize)
clientStatEntry.DataSize += int64(dealInfo.Proposal.PieceSize)
grandTotals.seenProvider[dealInfo.Proposal.Provider] = true
projStatEntry.dataPerProvider[dealInfo.Proposal.Provider] += int64(dealInfo.Proposal.PieceSize)
clientStatEntry.providers[dealInfo.Proposal.Provider] = true
grandTotals.seenPieceCid[dealInfo.Proposal.PieceCID] = true
projStatEntry.cidDeals[dealInfo.Proposal.PieceCID]++
clientStatEntry.cids[dealInfo.Proposal.PieceCID] = true
grandTotals.TotalDeals++
projStatEntry.NumDeals++
clientStatEntry.NumDeals++
if dealInfo.Proposal.VerifiedDeal {
grandTotals.FilplusTotalDeals++
grandTotals.FilplusTotalBytes += int64(dealInfo.Proposal.PieceSize)
}
payloadCid := "unknown"
if c, err := cid.Parse(dealInfo.Proposal.Label); err == nil {
payloadCid = c.String()
}
projDealLists[projID] = append(projDealLists[projID], &individualDeal{
DealID: dealID,
ProjectID: projID,
Client: clientAddr.String(),
MinerID: dealInfo.Proposal.Provider.String(),
PayloadCID: payloadCid,
PaddedSize: int64(dealInfo.Proposal.PieceSize),
DealStartEpoch: int64(dealInfo.State.SectorStartEpoch),
})
}
//
// Write out per-project deal lists
for proj, dl := range projDealLists {
err := func() error {
outListFd, err := os.Create(fmt.Sprintf(outDirName+"/deals_list_%s.json", proj))
if err != nil {
return err
}
defer outListFd.Close() //nolint:errcheck
ridiculousLintMandatedRebind := dl
sort.Slice(dl, func(i, j int) bool {
return ridiculousLintMandatedRebind[j].PaddedSize < ridiculousLintMandatedRebind[i].PaddedSize
})
if err := json.NewEncoder(outListFd).Encode(
dealListOutput{
Epoch: int64(head.Height()),
Endpoint: "DEAL_LIST",
Payload: dl,
},
); err != nil {
return err
}
return nil
}()
if err != nil {
return err
}
}
//
// write out basic_stats.json and unfiltered_basic_stats.json
for _, st := range []*competitionTotal{&grandTotals, &unfilteredGrandTotals} {
st.UniqueCids = len(st.seenPieceCid)
st.UniqueClients = len(st.seenClient)
st.UniqueProviders = len(st.seenProvider)
if st.seenProject != nil {
st.UniqueProjects = len(st.seenProject)
}
}
if err := json.NewEncoder(outBasicStatsFd).Encode(
competitionTotalOutput{
Epoch: int64(head.Height()),
Endpoint: "COMPETITION_TOTALS",
Payload: grandTotals,
},
); err != nil {
return err
}
if err := json.NewEncoder(outUnfilteredStatsFd).Encode(
competitionTotalOutput{
Epoch: int64(head.Height()),
Endpoint: "NETWORK_WIDE_TOTALS",
Payload: unfilteredGrandTotals,
},
); err != nil {
return err
}
//
// write out client_stats.json
for _, ps := range projStats {
ps.NumCids = len(ps.cidDeals)
ps.NumProviders = len(ps.dataPerProvider)
for _, dealsForCid := range ps.cidDeals {
if ps.HighestCidDealCount < dealsForCid {
ps.HighestCidDealCount = dealsForCid
}
}
for _, dataForProvider := range ps.dataPerProvider {
if ps.DataSizeMaxProvider < dataForProvider {
ps.DataSizeMaxProvider = dataForProvider
}
}
for _, cs := range ps.ClientStats {
cs.NumCids = len(cs.cids)
cs.NumProviders = len(cs.providers)
}
}
if err := json.NewEncoder(outClientStatsFd).Encode(
projectAggregateStatsOutput{
Epoch: int64(head.Height()),
Endpoint: "PROJECT_DEAL_STATS",
Payload: projStats,
},
); err != nil {
return err
}
return nil
},
}

114
cmd/lotus-shed/storage-stats.go Normal file
View File

@ -0,0 +1,114 @@
package main
import (
"encoding/json"
"os"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/ipfs/go-cid"
"github.com/urfave/cli/v2"
)
// How many epochs back to look at for dealstats
var defaultEpochLookback = abi.ChainEpoch(10)
type networkTotalsOutput struct {
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload networkTotals `json:"payload"`
}
type networkTotals struct {
UniqueCids int `json:"total_unique_cids"`
UniqueProviders int `json:"total_unique_providers"`
UniqueClients int `json:"total_unique_clients"`
TotalDeals int `json:"total_num_deals"`
TotalBytes int64 `json:"total_stored_data_size"`
FilplusTotalDeals int `json:"filplus_total_num_deals"`
FilplusTotalBytes int64 `json:"filplus_total_stored_data_size"`
seenClient map[address.Address]bool
seenProvider map[address.Address]bool
seenPieceCid map[cid.Cid]bool
}
var storageStatsCmd = &cli.Command{
Name: "storage-stats",
Usage: "Translates current lotus state into a json summary suitable for driving https://storage.filecoin.io/",
Flags: []cli.Flag{
&cli.Int64Flag{
Name: "height",
},
},
Action: func(cctx *cli.Context) error {
ctx := lcli.ReqContext(cctx)
api, apiCloser, err := lcli.GetFullNodeAPI(cctx)
if err != nil {
return err
}
defer apiCloser()
head, err := api.ChainHead(ctx)
if err != nil {
return err
}
requestedHeight := cctx.Int64("height")
if requestedHeight > 0 {
head, err = api.ChainGetTipSetByHeight(ctx, abi.ChainEpoch(requestedHeight), head.Key())
} else {
head, err = api.ChainGetTipSetByHeight(ctx, head.Height()-defaultEpochLookback, head.Key())
}
if err != nil {
return err
}
netTotals := networkTotals{
seenClient: make(map[address.Address]bool),
seenProvider: make(map[address.Address]bool),
seenPieceCid: make(map[cid.Cid]bool),
}
deals, err := api.StateMarketDeals(ctx, head.Key())
if err != nil {
return err
}
for _, dealInfo := range deals {
// Only count deals that have properly started, not past/future ones
// https://github.com/filecoin-project/specs-actors/blob/v0.9.9/actors/builtin/market/deal.go#L81-L85
// Bail on 0 as well in case SectorStartEpoch is uninitialized due to some bug
if dealInfo.State.SectorStartEpoch <= 0 ||
dealInfo.State.SectorStartEpoch > head.Height() {
continue
}
netTotals.seenClient[dealInfo.Proposal.Client] = true
netTotals.TotalBytes += int64(dealInfo.Proposal.PieceSize)
netTotals.seenProvider[dealInfo.Proposal.Provider] = true
netTotals.seenPieceCid[dealInfo.Proposal.PieceCID] = true
netTotals.TotalDeals++
if dealInfo.Proposal.VerifiedDeal {
netTotals.FilplusTotalDeals++
netTotals.FilplusTotalBytes += int64(dealInfo.Proposal.PieceSize)
}
}
netTotals.UniqueCids = len(netTotals.seenPieceCid)
netTotals.UniqueClients = len(netTotals.seenClient)
netTotals.UniqueProviders = len(netTotals.seenProvider)
return json.NewEncoder(os.Stdout).Encode(
networkTotalsOutput{
Epoch: int64(head.Height()),
Endpoint: "NETWORK_WIDE_TOTALS",
Payload: netTotals,
},
)
},
}

14
cmd/lotus-storage-miner/actor.go
View File

@ -420,6 +420,7 @@ var actorControlList = &cli.Command{
commit := map[address.Address]struct{}{}
precommit := map[address.Address]struct{}{}
terminate := map[address.Address]struct{}{}
post := map[address.Address]struct{}{}
for _, ca := range mi.ControlAddresses {
@ -446,6 +447,16 @@ var actorControlList = &cli.Command{
commit[ca] = struct{}{}
}
for _, ca := range ac.TerminateControl {
ca, err := api.StateLookupID(ctx, ca, types.EmptyTSK)
if err != nil {
return err
}
delete(post, ca)
terminate[ca] = struct{}{}
}
printKey := func(name string, a address.Address) {
b, err := api.WalletBalance(ctx, a)
if err != nil {
@ -487,6 +498,9 @@ var actorControlList = &cli.Command{
if _, ok := commit[a]; ok {
uses = append(uses, color.BlueString("commit"))
}
if _, ok := terminate[a]; ok {
uses = append(uses, color.YellowString("terminate"))
}
tw.Write(map[string]interface{}{
"name": name,

3
cmd/lotus-storage-miner/actor_test.go
View File

@ -17,6 +17,7 @@ import (
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/test"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/actors/policy"
@ -70,7 +71,7 @@ func TestWorkerKeyChange(t *testing.T) {
"testnode-storage": sn[0],
}
app.Writer = output
build.RunningNodeType = build.NodeMiner
api.RunningNodeType = api.NodeMiner
fs := flag.NewFlagSet("", flag.ContinueOnError)
for _, f := range cmd.Flags {

4
cmd/lotus-storage-miner/allinfo_test.go
View File

@ -11,8 +11,8 @@ import (
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/test"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/actors/policy"
"github.com/filecoin-project/lotus/lib/lotuslog"
"github.com/filecoin-project/lotus/node/repo"
@ -55,7 +55,7 @@ func TestMinerAllInfo(t *testing.T) {
"testnode-full": n[0],
"testnode-storage": sn[0],
}
build.RunningNodeType = build.NodeMiner
api.RunningNodeType = api.NodeMiner
cctx := cli.NewContext(app, flag.NewFlagSet("", flag.ContinueOnError), nil)

40
cmd/lotus-storage-miner/info_all.go
View File

@ -35,80 +35,80 @@ var infoAllCmd = &cli.Command{
fmt.Println("#: Version")
if err := lcli.VersionCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Miner Info")
if err := infoCmdAct(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
// Verbose info
fmt.Println("\n#: Storage List")
if err := storageListCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Worker List")
if err := sealingWorkersCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: PeerID")
if err := lcli.NetId.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Listen Addresses")
if err := lcli.NetListen.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Reachability")
if err := lcli.NetReachability.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
// Very Verbose info
fmt.Println("\n#: Peers")
if err := lcli.NetPeers.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Sealing Jobs")
if err := sealingJobsCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Sched Diag")
if err := sealingSchedDiagCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Storage Ask")
if err := getAskCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Storage Deals")
if err := dealsListCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Retrieval Deals")
if err := retrievalDealsListCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Sector List")
if err := sectorsListCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
fmt.Println("\n#: Sector Refs")
if err := sectorsRefsCmd.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
// Very Very Verbose info
@ -116,7 +116,7 @@ var infoAllCmd = &cli.Command{
list, err := nodeApi.SectorsList(ctx)
if err != nil {
return err
fmt.Println("ERROR: ", err)
}
sort.Slice(list, func(i, j int) bool {
@ -129,11 +129,11 @@ var infoAllCmd = &cli.Command{
fs := &flag.FlagSet{}
for _, f := range sectorsStatusCmd.Flags {
if err := f.Apply(fs); err != nil {
return err
fmt.Println("ERROR: ", err)
}
}
if err := fs.Parse([]string{"--log", "--on-chain-info", fmt.Sprint(s)}); err != nil {
return err
fmt.Println("ERROR: ", err)
}
if err := sectorsStatusCmd.Action(cli.NewContext(cctx.App, fs, cctx)); err != nil {
@ -144,7 +144,7 @@ var infoAllCmd = &cli.Command{
fs = &flag.FlagSet{}
if err := fs.Parse([]string{fmt.Sprint(s)}); err != nil {
return err
fmt.Println("ERROR: ", err)
}
if err := storageFindCmd.Action(cli.NewContext(cctx.App, fs, cctx)); err != nil {
@ -155,7 +155,7 @@ var infoAllCmd = &cli.Command{
if !_test {
fmt.Println("\n#: Goroutines")
if err := lcli.PprofGoroutines.Action(cctx); err != nil {
return err
fmt.Println("ERROR: ", err)
}
}

4
cmd/lotus-storage-miner/init.go
View File

@ -186,8 +186,8 @@ var initCmd = &cli.Command{
return err
}
if !v.APIVersion.EqMajorMinor(build.FullAPIVersion) {
return xerrors.Errorf("Remote API version didn't match (expected %s, remote %s)", build.FullAPIVersion, v.APIVersion)
if !v.APIVersion.EqMajorMinor(lapi.FullAPIVersion) {
return xerrors.Errorf("Remote API version didn't match (expected %s, remote %s)", lapi.FullAPIVersion, v.APIVersion)
}
log.Info("Initializing repo")

5
cmd/lotus-storage-miner/init_restore.go
View File

@ -18,6 +18,7 @@ import (
paramfetch "github.com/filecoin-project/go-paramfetch"
"github.com/filecoin-project/go-state-types/big"
lapi "github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/types"
lcli "github.com/filecoin-project/lotus/cli"
@ -68,8 +69,8 @@ var initRestoreCmd = &cli.Command{
return err
}
if !v.APIVersion.EqMajorMinor(build.FullAPIVersion) {
return xerrors.Errorf("Remote API version didn't match (expected %s, remote %s)", build.FullAPIVersion, v.APIVersion)
if !v.APIVersion.EqMajorMinor(lapi.FullAPIVersion) {
return xerrors.Errorf("Remote API version didn't match (expected %s, remote %s)", lapi.FullAPIVersion, v.APIVersion)
}
if !cctx.Bool("nosync") {

2
cmd/lotus-storage-miner/main.go
View File

@ -26,7 +26,7 @@ const FlagMinerRepo = "miner-repo"
const FlagMinerRepoDeprecation = "storagerepo"
func main() {
build.RunningNodeType = build.NodeMiner
api.RunningNodeType = api.NodeMiner
lotuslog.SetupLogLevels()

18
cmd/lotus-storage-miner/run.go
View File

@ -13,6 +13,7 @@ import (
"github.com/multiformats/go-multiaddr"
manet "github.com/multiformats/go-multiaddr/net"
"github.com/urfave/cli/v2"
"go.opencensus.io/stats"
"go.opencensus.io/stats/view"
"go.opencensus.io/tag"
"golang.org/x/xerrors"
@ -68,14 +69,20 @@ var runCmd = &cli.Command{
return xerrors.Errorf("getting full node api: %w", err)
}
defer ncloser()
ctx := lcli.DaemonContext(cctx)
ctx, _ := tag.New(lcli.DaemonContext(cctx),
tag.Insert(metrics.Version, build.BuildVersion),
tag.Insert(metrics.Commit, build.CurrentCommit),
tag.Insert(metrics.NodeType, "miner"),
)
// Register all metric views
if err := view.Register(
metrics.DefaultViews...,
if err = view.Register(
metrics.MinerNodeViews...,
); err != nil {
log.Fatalf("Cannot register the view: %v", err)
}
// Set the metric to one so it is published to the exporter
stats.Record(ctx, metrics.LotusInfo.M(1))
v, err := nodeApi.Version(ctx)
if err != nil {
@ -88,8 +95,8 @@ var runCmd = &cli.Command{
}
}
if v.APIVersion != build.FullAPIVersion {
return xerrors.Errorf("lotus-daemon API version doesn't match: expected: %s", api.Version{APIVersion: build.FullAPIVersion})
if v.APIVersion != api.FullAPIVersion {
return xerrors.Errorf("lotus-daemon API version doesn't match: expected: %s", api.APIVersion{APIVersion: api.FullAPIVersion})
}
log.Info("Checking full node sync status")
@ -162,6 +169,7 @@ var runCmd = &cli.Command{
mux.Handle("/rpc/v0", rpcServer)
mux.PathPrefix("/remote").HandlerFunc(minerapi.(*impl.StorageMinerAPI).ServeRemote)
mux.Handle("/debug/metrics", metrics.Exporter())
mux.PathPrefix("/").Handler(http.DefaultServeMux) // pprof
ah := &auth.Handler{

98
cmd/lotus-storage-miner/sectors.go
View File

@ -13,10 +13,13 @@ import (
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-bitfield"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/big"
miner0 "github.com/filecoin-project/specs-actors/actors/builtin/miner"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/chain/actors"
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
"github.com/filecoin-project/lotus/chain/actors/policy"
"github.com/filecoin-project/lotus/chain/types"
@ -35,6 +38,7 @@ var sectorsCmd = &cli.Command{
sectorsRefsCmd,
sectorsUpdateCmd,
sectorsPledgeCmd,
sectorsExtendCmd,
sectorsTerminateCmd,
sectorsRemoveCmd,
sectorsMarkForUpgradeCmd,
@ -410,6 +414,100 @@ var sectorsRefsCmd = &cli.Command{
},
}
var sectorsExtendCmd = &cli.Command{
Name: "extend",
Usage: "Extend sector expiration",
ArgsUsage: "<sectorNumbers...>",
Flags: []cli.Flag{
&cli.Int64Flag{
Name: "new-expiration",
Usage: "new expiration epoch",
Required: true,
},
&cli.StringFlag{},
},
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
if err != nil {
return err
}
defer closer()
api, nCloser, err := lcli.GetFullNodeAPI(cctx)
if err != nil {
return err
}
defer nCloser()
ctx := lcli.ReqContext(cctx)
if !cctx.Args().Present() {
return xerrors.Errorf("must pass at least one sector number")
}
maddr, err := nodeApi.ActorAddress(ctx)
if err != nil {
return xerrors.Errorf("getting miner actor address: %w", err)
}
sectors := map[miner.SectorLocation][]uint64{}
for i, s := range cctx.Args().Slice() {
id, err := strconv.ParseUint(s, 10, 64)
if err != nil {
return xerrors.Errorf("could not parse sector %d: %w", i, err)
}
p, err := api.StateSectorPartition(ctx, maddr, abi.SectorNumber(id), types.EmptyTSK)
if err != nil {
return xerrors.Errorf("getting sector location for sector %d: %w", id, err)
}
if p == nil {
return xerrors.Errorf("sector %d not found in any partition", id)
}
sectors[*p] = append(sectors[*p], id)
}
params := &miner0.ExtendSectorExpirationParams{}
for l, numbers := range sectors {
params.Extensions = append(params.Extensions, miner0.ExpirationExtension{
Deadline: l.Deadline,
Partition: l.Partition,
Sectors: bitfield.NewFromSet(numbers),
NewExpiration: abi.ChainEpoch(cctx.Int64("new-expiration")),
})
}
sp, err := actors.SerializeParams(params)
if err != nil {
return xerrors.Errorf("serializing params: %w", err)
}
mi, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
if err != nil {
return xerrors.Errorf("getting miner info: %w", err)
}
smsg, err := api.MpoolPushMessage(ctx, &types.Message{
From: mi.Worker,
To: maddr,
Method: miner.Methods.ExtendSectorExpiration,
Value: big.Zero(),
Params: sp,
}, nil)
if err != nil {
return xerrors.Errorf("mpool push message: %w", err)
}
fmt.Println(smsg.Cid())
return nil
},
}
var sectorsTerminateCmd = &cli.Command{
Name: "terminate",
Usage: "Terminate sector on-chain then remove (WARNING: This means losing power and collateral for the removed sector)",

94
cmd/lotus-storage-miner/storage.go
View File

@ -12,6 +12,7 @@ import (
"strings"
"time"
"github.com/docker/go-units"
"github.com/fatih/color"
"github.com/google/uuid"
"github.com/mitchellh/go-homedir"
@ -88,6 +89,10 @@ over time
Name: "store",
Usage: "(for init) use path for long-term storage",
},
&cli.StringFlag{
Name: "max-storage",
Usage: "(for init) limit storage space for sectors (expensive for very large paths!)",
},
},
Action: func(cctx *cli.Context) error {
nodeApi, closer, err := lcli.GetStorageMinerAPI(cctx)
@ -121,11 +126,20 @@ over time
return err
}
var maxStor int64
if cctx.IsSet("max-storage") {
maxStor, err = units.RAMInBytes(cctx.String("max-storage"))
if err != nil {
return xerrors.Errorf("parsing max-storage: %w", err)
}
}
cfg := &stores.LocalStorageMeta{
ID: stores.ID(uuid.New().String()),
Weight: cctx.Uint64("weight"),
CanSeal: cctx.Bool("seal"),
CanStore: cctx.Bool("store"),
ID: stores.ID(uuid.New().String()),
Weight: cctx.Uint64("weight"),
CanSeal: cctx.Bool("seal"),
CanStore: cctx.Bool("store"),
MaxStorage: uint64(maxStor),
}
if !(cfg.CanStore || cfg.CanSeal) {
@ -220,26 +234,66 @@ var storageListCmd = &cli.Command{
}
ping := time.Now().Sub(pingStart)
usedPercent := (st.Capacity - st.Available) * 100 / st.Capacity
percCol := color.FgGreen
switch {
case usedPercent > 98:
percCol = color.FgRed
case usedPercent > 90:
percCol = color.FgYellow
safeRepeat := func(s string, count int) string {
if count < 0 {
return ""
}
return strings.Repeat(s, count)
}
var barCols = int64(50)
set := (st.Capacity - st.Available) * barCols / st.Capacity
used := (st.Capacity - (st.Available + st.Reserved)) * barCols / st.Capacity
reserved := set - used
bar := strings.Repeat("#", int(used)) + strings.Repeat("*", int(reserved)) + strings.Repeat(" ", int(barCols-set))
fmt.Printf("\t[%s] %s/%s %s\n", color.New(percCol).Sprint(bar),
types.SizeStr(types.NewInt(uint64(st.Capacity-st.Available))),
types.SizeStr(types.NewInt(uint64(st.Capacity))),
color.New(percCol).Sprintf("%d%%", usedPercent))
// filesystem use bar
{
usedPercent := (st.Capacity - st.FSAvailable) * 100 / st.Capacity
percCol := color.FgGreen
switch {
case usedPercent > 98:
percCol = color.FgRed
case usedPercent > 90:
percCol = color.FgYellow
}
set := (st.Capacity - st.FSAvailable) * barCols / st.Capacity
used := (st.Capacity - (st.FSAvailable + st.Reserved)) * barCols / st.Capacity
reserved := set - used
bar := safeRepeat("#", int(used)) + safeRepeat("*", int(reserved)) + safeRepeat(" ", int(barCols-set))
desc := ""
if st.Max > 0 {
desc = " (filesystem)"
}
fmt.Printf("\t[%s] %s/%s %s%s\n", color.New(percCol).Sprint(bar),
types.SizeStr(types.NewInt(uint64(st.Capacity-st.FSAvailable))),
types.SizeStr(types.NewInt(uint64(st.Capacity))),
color.New(percCol).Sprintf("%d%%", usedPercent), desc)
}
// optional configured limit bar
if st.Max > 0 {
usedPercent := st.Used * 100 / st.Max
percCol := color.FgGreen
switch {
case usedPercent > 98:
percCol = color.FgRed
case usedPercent > 90:
percCol = color.FgYellow
}
set := st.Used * barCols / st.Max
used := (st.Used + st.Reserved) * barCols / st.Max
reserved := set - used
bar := safeRepeat("#", int(used)) + safeRepeat("*", int(reserved)) + safeRepeat(" ", int(barCols-set))
fmt.Printf("\t[%s] %s/%s %s (limit)\n", color.New(percCol).Sprint(bar),
types.SizeStr(types.NewInt(uint64(st.Used))),
types.SizeStr(types.NewInt(uint64(st.Max))),
color.New(percCol).Sprintf("%d%%", usedPercent))
}
fmt.Printf("\t%s; %s; %s; Reserved: %s\n",
color.YellowString("Unsealed: %d", cnt[0]),
color.GreenString("Sealed: %d", cnt[1]),

30
cmd/lotus/daemon.go
View File

@ -192,7 +192,20 @@ var DaemonCmd = &cli.Command{
return fmt.Errorf("unrecognized profile type: %q", profile)
}
ctx, _ := tag.New(context.Background(), tag.Insert(metrics.Version, build.BuildVersion), tag.Insert(metrics.Commit, build.CurrentCommit))
ctx, _ := tag.New(context.Background(),
tag.Insert(metrics.Version, build.BuildVersion),
tag.Insert(metrics.Commit, build.CurrentCommit),
tag.Insert(metrics.NodeType, "chain"),
)
// Register all metric views
if err = view.Register(
metrics.ChainNodeViews...,
); err != nil {
log.Fatalf("Cannot register the view: %v", err)
}
// Set the metric to one so it is published to the exporter
stats.Record(ctx, metrics.LotusInfo.M(1))
{
dir, err := homedir.Expand(cctx.String("repo"))
if err != nil {
@ -300,11 +313,12 @@ var DaemonCmd = &cli.Command{
stop, err := node.New(ctx,
node.FullAPI(&api, node.Lite(isLite)),
node.Override(new(dtypes.Bootstrapper), isBootstrapper),
node.Override(new(dtypes.ShutdownChan), shutdownChan),
node.Online(),
node.Repo(r),
node.Override(new(dtypes.Bootstrapper), isBootstrapper),
node.Override(new(dtypes.ShutdownChan), shutdownChan),
genesis,
liteModeDeps,
@ -332,16 +346,6 @@ var DaemonCmd = &cli.Command{
}
}
// Register all metric views
if err = view.Register(
metrics.DefaultViews...,
); err != nil {
log.Fatalf("Cannot register the view: %v", err)
}
// Set the metric to one so it is published to the exporter
stats.Record(ctx, metrics.LotusInfo.M(1))
endpoint, err := r.APIEndpoint()
if err != nil {
return xerrors.Errorf("getting api endpoint: %w", err)

3
cmd/lotus/main.go
View File

@ -6,6 +6,7 @@ import (
"github.com/urfave/cli/v2"
"go.opencensus.io/trace"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/filecoin-project/lotus/lib/lotuslog"
@ -16,7 +17,7 @@ import (
var AdvanceBlockCmd *cli.Command
func main() {
build.RunningNodeType = build.NodeFull
api.RunningNodeType = api.NodeFull
lotuslog.SetupLogLevels()

22
cmd/lotus/rpc.go
View File

@ -15,12 +15,9 @@ import (
logging "github.com/ipfs/go-log/v2"
"github.com/multiformats/go-multiaddr"
manet "github.com/multiformats/go-multiaddr/net"
promclient "github.com/prometheus/client_golang/prometheus"
"go.opencensus.io/tag"
"golang.org/x/xerrors"
"contrib.go.opencensus.io/exporter/prometheus"
"github.com/filecoin-project/go-jsonrpc"
"github.com/filecoin-project/go-jsonrpc/auth"
@ -40,6 +37,7 @@ func serveRPC(a api.FullNode, stop node.StopFunc, addr multiaddr.Multiaddr, shut
}
rpcServer := jsonrpc.NewServer(serverOptions...)
rpcServer.Register("Filecoin", apistruct.PermissionedFullAPI(metrics.MetricedFullAPI(a)))
rpcServer.AliasMethod("rpc.discover", "Filecoin.Discover")
ah := &auth.Handler{
Verify: a.AuthVerify,
@ -55,23 +53,7 @@ func serveRPC(a api.FullNode, stop node.StopFunc, addr multiaddr.Multiaddr, shut
http.Handle("/rest/v0/import", importAH)
// Prometheus globals are exposed as interfaces, but the prometheus
// OpenCensus exporter expects a concrete *Registry. The concrete type of
// the globals are actually *Registry, so we downcast them, staying
// defensive in case things change under the hood.
registry, ok := promclient.DefaultRegisterer.(*promclient.Registry)
if !ok {
log.Warnf("failed to export default prometheus registry; some metrics will be unavailable; unexpected type: %T", promclient.DefaultRegisterer)
}
exporter, err := prometheus.NewExporter(prometheus.Options{
Registry: registry,
Namespace: "lotus",
})
if err != nil {
log.Fatalf("could not create the prometheus stats exporter: %v", err)
}
http.Handle("/debug/metrics", exporter)
http.Handle("/debug/metrics", metrics.Exporter())
http.Handle("/debug/pprof-set/block", handleFractionOpt("BlockProfileRate", runtime.SetBlockProfileRate))
http.Handle("/debug/pprof-set/mutex", handleFractionOpt("MutexProfileFraction",
func(x int) { runtime.SetMutexProfileFraction(x) },

2
cmd/tvx/simulate.go
View File

@ -154,7 +154,7 @@ func runSimulateCmd(_ *cli.Context) error {
version, err := FullAPI.Version(ctx)
if err != nil {
log.Printf("failed to get node version: %s; falling back to unknown", err)
version = api.Version{}
version = api.APIVersion{}
}
nv, err := FullAPI.StateNetworkVersion(ctx, ts.Key())

74
documentation/en/api-methods-miner.md
View File

@ -1,6 +1,7 @@
# Groups
* [](#)
* [Closing](#Closing)
* [Discover](#Discover)
* [Session](#Session)
* [Shutdown](#Shutdown)
* [Version](#Version)
@ -69,6 +70,7 @@
* [NetConnectedness](#NetConnectedness)
* [NetDisconnect](#NetDisconnect)
* [NetFindPeer](#NetFindPeer)
* [NetPeerInfo](#NetPeerInfo)
* [NetPeers](#NetPeers)
* [NetPubsubScores](#NetPubsubScores)
* [Pieces](#Pieces)
@ -141,6 +143,25 @@ Inputs: `null`
Response: `{}`
### Discover
Perms: read
Inputs: `null`
Response:
```json
{
"info": {
"title": "Lotus RPC API",
"version": "1.2.1/generated=2020-11-22T08:22:42-06:00"
},
"methods": [],
"openrpc": "1.2.6"
}
```
### Session
@ -199,7 +220,9 @@ Response:
{
"PreCommitControl": null,
"CommitControl": null,
"TerminateControl": null
"TerminateControl": null,
"DisableOwnerFallback": true,
"DisableWorkerFallback": true
}
```
@ -1045,6 +1068,38 @@ Response:
}
```
### NetPeerInfo
Perms: read
Inputs:
```json
[
"12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"
]
```
Response:
```json
{
"ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",
"Agent": "string value",
"Addrs": null,
"Protocols": null,
"ConnMgrMeta": {
"FirstSeen": "0001-01-01T00:00:00Z",
"Value": 123,
"Tags": {
"name": 42
},
"Conns": {
"name": "2021-03-08T22:52:18Z"
}
}
}
```
### NetPeers
@ -1779,13 +1834,17 @@ Inputs:
"ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8",
"URLs": null,
"Weight": 42,
"MaxStorage": 42,
"CanSeal": true,
"CanStore": true
},
{
"Capacity": 9,
"Available": 9,
"Reserved": 9
"FSAvailable": 9,
"Reserved": 9,
"Max": 9,
"Used": 9
}
]
```
@ -1885,6 +1944,7 @@ Response:
"ID": "76f1988b-ef30-4d7e-b3ec-9a627f4ba5a8",
"URLs": null,
"Weight": 42,
"MaxStorage": 42,
"CanSeal": true,
"CanStore": true
}
@ -1956,7 +2016,10 @@ Inputs:
"Stat": {
"Capacity": 9,
"Available": 9,
"Reserved": 9
"FSAvailable": 9,
"Reserved": 9,
"Max": 9,
"Used": 9
},
"Err": "string value"
}
@ -1982,7 +2045,10 @@ Response:
{
"Capacity": 9,
"Available": 9,
"Reserved": 9
"FSAvailable": 9,
"Reserved": 9,
"Max": 9,
"Used": 9
}
```

180
documentation/en/api-methods.md
View File

@ -1,6 +1,7 @@
# Groups
* [](#)
* [Closing](#Closing)
* [Discover](#Discover)
* [Session](#Session)
* [Shutdown](#Shutdown)
* [Version](#Version)
@ -121,6 +122,7 @@
* [NetConnectedness](#NetConnectedness)
* [NetDisconnect](#NetDisconnect)
* [NetFindPeer](#NetFindPeer)
* [NetPeerInfo](#NetPeerInfo)
* [NetPeers](#NetPeers)
* [NetPubsubScores](#NetPubsubScores)
* [Paych](#Paych)
@ -225,6 +227,25 @@ Inputs: `null`
Response: `{}`
### Discover
Perms: read
Inputs: `null`
Response:
```json
{
"info": {
"title": "Lotus RPC API",
"version": "1.2.1/generated=2020-11-22T08:22:42-06:00"
},
"methods": [],
"openrpc": "1.2.6"
}
```
### Session
@ -424,6 +445,17 @@ Response:
### ChainGetBlockMessages
ChainGetBlockMessages returns messages stored in the specified block.
Note: If there are multiple blocks in a tipset, it's likely that some
messages will be duplicated. It's also possible for blocks in a tipset to have
different messages from the same sender at the same nonce. When that happens,
only the first message (in a block with lowest ticket) will be considered
for execution
NOTE: THIS METHOD SHOULD ONLY BE USED FOR GETTING MESSAGES IN A SPECIFIC BLOCK
DO NOT USE THIS METHOD TO GET MESSAGES INCLUDED IN A TIPSET
Use ChainGetParentMessages, which will perform correct message deduplication
Perms: read
@ -539,7 +571,8 @@ Response: `null`
### ChainGetParentReceipts
ChainGetParentReceipts returns receipts for messages in parent tipset of
the specified block.
the specified block. The receipts in the list returned is one-to-one with the
messages returned by a call to ChainGetParentMessages with the same blockCid.
Perms: read
@ -2887,6 +2920,38 @@ Response:
}
```
### NetPeerInfo
Perms: read
Inputs:
```json
[
"12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"
]
```
Response:
```json
{
"ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",
"Agent": "string value",
"Addrs": null,
"Protocols": null,
"ConnMgrMeta": {
"FirstSeen": "0001-01-01T00:00:00Z",
"Value": 123,
"Tags": {
"name": 42
},
"Conns": {
"name": "2021-03-08T22:52:18Z"
}
}
}
```
### NetPeers
@ -3532,6 +3597,36 @@ Response: `"0"`
StateCompute is a flexible command that applies the given messages on the given tipset.
The messages are run as though the VM were at the provided height.
When called, StateCompute will:
- Load the provided tipset, or use the current chain head if not provided
- Compute the tipset state of the provided tipset on top of the parent state
- (note that this step runs before vmheight is applied to the execution)
- Execute state upgrade if any were scheduled at the epoch, or in null
blocks preceding the tipset
- Call the cron actor on null blocks preceding the tipset
- For each block in the tipset
- Apply messages in blocks in the specified
- Award block reward by calling the reward actor
- Call the cron actor for the current epoch
- If the specified vmheight is higher than the current epoch, apply any
needed state upgrades to the state
- Apply the specified messages to the state
The vmheight parameter sets VM execution epoch, and can be used to simulate
message execution in different network versions. If the specified vmheight
epoch is higher than the epoch of the specified tipset, any state upgrades
until the vmheight will be executed on the state before applying messages
specified by the user.
Note that the initial tipset state computation is not affected by the
vmheight parameter - only the messages in the `apply` set are
If the caller wants to simply compute the state, vmheight should be set to
the epoch of the specified tipset.
Messages in the `apply` parameter must have the correct nonces, and gas
values set.
Perms: read
@ -3653,7 +3748,15 @@ Response:
```
### StateGetReceipt
StateGetReceipt returns the message receipt for the given message
StateGetReceipt returns the message receipt for the given message or for a
matching gas-repriced replacing message
NOTE: If the requested message was replaced, this method will return the receipt
for the replacing message - if the caller needs the receipt for exactly the
requested message, use StateSearchMsg().Receipt, and check that MsgLookup.Message
is matching the requested CID
DEPRECATED: Use StateSearchMsg, this method won't be supported in v1 API
Perms: read
@ -4417,7 +4520,22 @@ Response:
### StateReplay
StateReplay replays a given message, assuming it was included in a block in the specified tipset.
If no tipset key is provided, the appropriate tipset is looked up.
If a tipset key is provided, and a replacing message is found on chain,
the method will return an error saying that the message wasn't found
If no tipset key is provided, the appropriate tipset is looked up, and if
the message was gas-repriced, the on-chain message will be replayed - in
that case the returned InvocResult.MsgCid will not match the Cid param
If the caller wants to ensure that exactly the requested message was executed,
they MUST check that InvocResult.MsgCid is equal to the provided Cid.
Without this check both the requested and original message may appear as
successfully executed on-chain, which may look like a double-spend.
A replacing message is a message with a different CID, any of Gas values, and
different signature, but with all other parameters matching (source/destination,
nonce, params, etc.)
Perms: read
@ -4511,6 +4629,20 @@ Response:
### StateSearchMsg
StateSearchMsg searches for a message in the chain, and returns its receipt and the tipset where it was executed
NOTE: If a replacing message is found on chain, this method will return
a MsgLookup for the replacing message - the MsgLookup.Message will be a different
CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
result of the execution of the replacing message.
If the caller wants to ensure that exactly the requested message was executed,
they MUST check that MsgLookup.Message is equal to the provided 'cid'.
Without this check both the requested and original message may appear as
successfully executed on-chain, which may look like a double-spend.
A replacing message is a message with a different CID, any of Gas values, and
different signature, but with all other parameters matching (source/destination,
nonce, params, etc.)
Perms: read
@ -4550,6 +4682,20 @@ Response:
### StateSearchMsgLimited
StateSearchMsgLimited looks back up to limit epochs in the chain for a message, and returns its receipt and the tipset where it was executed
NOTE: If a replacing message is found on chain, this method will return
a MsgLookup for the replacing message - the MsgLookup.Message will be a different
CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
result of the execution of the replacing message.
If the caller wants to ensure that exactly the requested message was executed,
they MUST check that MsgLookup.Message is equal to the provided 'cid'.
Without this check both the requested and original message may appear as
successfully executed on-chain, which may look like a double-spend.
A replacing message is a message with a different CID, any of Gas values, and
different signature, but with all other parameters matching (source/destination,
nonce, params, etc.)
Perms: read
@ -4844,6 +4990,20 @@ Response: `"0"`
StateWaitMsg looks back in the chain for a message. If not found, it blocks until the
message arrives on chain, and gets to the indicated confidence depth.
NOTE: If a replacing message is found on chain, this method will return
a MsgLookup for the replacing message - the MsgLookup.Message will be a different
CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
result of the execution of the replacing message.
If the caller wants to ensure that exactly the requested message was executed,
they MUST check that MsgLookup.Message is equal to the provided 'cid'.
Without this check both the requested and original message may appear as
successfully executed on-chain, which may look like a double-spend.
A replacing message is a message with a different CID, any of Gas values, and
different signature, but with all other parameters matching (source/destination,
nonce, params, etc.)
Perms: read
@ -4886,6 +5046,20 @@ StateWaitMsgLimited looks back up to limit epochs in the chain for a message.
If not found, it blocks until the message arrives on chain, and gets to the
indicated confidence depth.
NOTE: If a replacing message is found on chain, this method will return
a MsgLookup for the replacing message - the MsgLookup.Message will be a different
CID than the one provided in the 'cid' param, MsgLookup.Receipt will contain the
result of the execution of the replacing message.
If the caller wants to ensure that exactly the requested message was executed,
they MUST check that MsgLookup.Message is equal to the provided 'cid'.
Without this check both the requested and original message may appear as
successfully executed on-chain, which may look like a double-spend.
A replacing message is a message with a different CID, any of Gas values, and
different signature, but with all other parameters matching (source/destination,
nonce, params, etc.)
Perms: read

4
extern/sector-storage/fr32/readers_test.go vendored
View File

@ -1,6 +1,7 @@
package fr32_test
import (
"bufio"
"bytes"
"io/ioutil"
"testing"
@ -25,7 +26,8 @@ func TestUnpadReader(t *testing.T) {
t.Fatal(err)
}
readered, err := ioutil.ReadAll(r)
// using bufio reader to make sure reads are big enough for the padreader - it can't handle small reads right now
readered, err := ioutil.ReadAll(bufio.NewReaderSize(r, 512))
if err != nil {
t.Fatal(err)
}

11
extern/sector-storage/fsutil/statfs.go vendored
View File

@ -1,7 +1,12 @@
package fsutil
type FsStat struct {
Capacity int64
Available int64 // Available to use for sector storage
Reserved int64
Capacity int64
Available int64 // Available to use for sector storage
FSAvailable int64 // Available in the filesystem
Reserved int64
// non-zero when storage has configured MaxStorage
Max int64
Used int64
}

6
extern/sector-storage/fsutil/statfs_unix.go vendored
View File

@ -15,7 +15,9 @@ func Statfs(path string) (FsStat, error) {
// force int64 to handle platform specific differences
//nolint:unconvert
return FsStat{
Capacity: int64(stat.Blocks) * int64(stat.Bsize),
Available: int64(stat.Bavail) * int64(stat.Bsize),
Capacity: int64(stat.Blocks) * int64(stat.Bsize),
Available: int64(stat.Bavail) * int64(stat.Bsize),
FSAvailable: int64(stat.Bavail) * int64(stat.Bsize),
}, nil
}

Some files were not shown because too many files have changed in this diff Show More