lotus/node/impl/storminer.go

725 lines
23 KiB
Go
Raw Normal View History

package impl
2019-07-24 00:58:31 +00:00
import (
2019-07-27 01:54:03 +00:00
"context"
"encoding/json"
"net/http"
"os"
"strconv"
"time"
2020-02-27 21:45:31 +00:00
2021-03-29 02:31:31 +00:00
"github.com/filecoin-project/lotus/chain/actors/builtin"
"github.com/filecoin-project/lotus/chain/gen"
OpenRPC Support (#5843) * main: init implement rpc.Discover RPC method This implement the basic functionality for the method over HTTP RPC. Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init example with go-openrpc-reflect lib Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * main: make variable name human-friendly Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init impl of go-openrp-reflect printing document Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * go.mod,go.sum: use go-openrpc-reflect and open-rpc/meta-schema hackforks This is for development only. Versions need to be bumped when they're ready for use as canonical remotes. Signed-off-by: meows <b5c6@protonmail.com> * main,openrpc,main: refactor openrpc supporting code to own package This eliminates code duplication. Signed-off-by: meows <b5c6@protonmail.com> * main: add rpc.Discover to openrpc document Signed-off-by: meows <b5c6@protonmail.com> * openrpc: fix rpc.discover method name casing Also fixes casing stuff for the rest of Filecoin. methods. Signed-off-by: meows <b5c6@protonmail.com> * Revert "main: add rpc.Discover to openrpc document" This reverts commit 116898efb10f33e405ac74acb1aa6daefcd46a62. * main: fix document creation method name This fixes an issue caused with the latest reverting commit. Signed-off-by: meows <b5c6@protonmail.com> * main,docgen,openrpc: refactor to share api parsing, etc as docgen exported stuff Signed-off-by: meows <b5c6@protonmail.com> Makefile: fix docgen refactoring for makefile use of command Signed-off-by: meows <b5c6@protonmail.com> * openrpc: add schema.examples to app reflector There are quite of few of these already registered for the docgen command, so it makes sense to use those! Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init method pairing examples Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump go.mod to use latest meta-schema and openrpc-reflect versions Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init SchemaType mapper function This function will handle the manual configurations for app-specific data types w/r/t their json schema representation. This is useful for cases where the reflect library is unable to provide a sufficient representation automatically. Provided in this commit is an initial implementation for the integerD type (assuming number are represented in the API as hexs), and a commonly used cid.Cid type. Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: tame dependencies by bumping etclabscore/go-openrpc-reflect This removes a problematic dependency on github.com/ethereum/go-ethereum, which was imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect tests. etclabscore/go-openrpc-reflect v0.0.36 has removed this dependency, so this commit is the result of bumping that version and then running 'go mod tidy' This is in response to a review at https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 Date: 2020-11-21 06:52:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: add 'miner' arg to openrpc gen cmd This allows the command to EITHER generate the doc for Full or Miner APIs. See comment for usage. Date: 2020-11-21 07:48:05-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen: add missing examples for Miner API Generating the Miner API OpenRPC doc (via 'go run ./api/openrpc/cmd miner') caused the example logic to panic because some types were missing. This commit adds those missing types, although I'm not an expert in the API so I can't suggest that the example values provided are ideal or well representative. Date: 2020-11-21 07:50:21-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json,build/openrpc/miner.json: add build/openrpc/[full/miner].json docs These will be used as static documents provided by the rpc.discover method. Date: 2020-11-21 07:51:39-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: init go-rice openrpc static assets Date: 2020-11-21 08:23:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: remove rpc.discover implementation from runtime plugin Instead of generating the doc on the fly, we're going to serve a static asset. Rel https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 This removes the runtime implementation from the RPC server construction. Date: 2020-11-21 08:41:20-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,common: add Discover(ctx) method to CommonAPI interface and structs Date: 2020-11-21 08:41:56-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: use rpc server method aliasing for rpc.discover This depends on a currently-forked change at filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210 which establishes this new method RPCServer.AliasMethod. This solves the problem that the OpenRPC spec says that the document should be served at the system extension-prefixed endpoing rpc.discover (not Filecoin.Discover). In fact, the document will be available at BOTH endpoints, but that duplicity is harmless. Date: 2020-11-21 09:18:26-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,build,common: rpc.discover: return json object instead of string Instead of casting the JSON asset from bytes to string, unmarshal it to a map[string]interface{} so the server will provide it as a JSON object. Date: 2020-11-21 09:27:11-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile: merge resolve: docsgen command path Date: 2020-11-22 07:19:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,main,docgen,openrpc: merge resolve: fix func exporteds, signatures Date: 2020-11-22 07:31:03-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: 'get get' auto-bumps version Date: 2020-11-22 07:31:44-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,docgen,main,build/openrpc: refactor openrpc documentation generation This creates Makefile command docsgen-openrpc-json, and refactors the docsgen command to generate both the markdown and openrpc json documents, redirecting the output of the openrpc json documentation to the build/openrpc/ directory, where those json files will be compiled as static assets via go-rice boxes. The api/openrpc/cmd now uses usage argumentation congruent to that of the docgen command (switching on API context). Date: 2020-11-22 08:01:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc: rename api/openrpc -> api/docgen-openrpc Renames the package as well. This is intended to parallel the existing docgen package and command namespacing. Date: 2020-11-22 10:34:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,docgen,build,build/openrpc: use typed Discover response Instead of using a map[string]interface{}, use a typed response for the Discover method implementation. This avoids having to set a docgen Example for the generic map[string]interface{} (as an openrpc document) which both pollutes the generic type and lacks useful information for the Discover method example. Date: 2020-11-22 08:31:16-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,build,main,impl: implement Discover method for Worker and StorageMiner APIs Methods return static compiled assets respective to the APIs. Date: 2020-11-22 08:57:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen_openrpc,build/openrpc: remove timestamping from openrpc doc info This should allow openrpc docs generated at different times to be equal. This is important because the CI (Circle) runs the docgen command and tests that the output and the source are unchanged (via git diff). Date: 2020-11-22 10:47:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc,main,build: fix lint issues Fixes goimports, staticcheck, golint issues. Date: 2020-11-22 11:06:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: fix: don't use an underscore in package name (golint) Date: 2020-11-22 11:07:53-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: fix: mod-tidy-check (run 'go mod tidy') Date: 2020-11-22 11:09:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc dep to latest This version includes the necessary RPCServer.AliasMethod method. Date: 2020-11-23 12:16:15-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,main,build,build/openrpc: init gzipped openrpc static docs Date: 2020-11-24 06:15:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: refactor gzip reading Date: 2020-11-24 06:18:34-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: add basic test for openrpc doc from static assets Date: 2020-11-24 06:30:23-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: handle reader Close error This keeps the errcheck linter happy. Date: 2020-11-24 06:33:14-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: run 'go mod tidy' Date: 2020-11-24 06:36:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: go mod tidy Tidying up after resolving the merge conflicts with master at go.mod Date: 2020-11-24 06:40:45-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc to latest This is a repeat of 76e6fd2, since the latest merge to master seems to have reverted this. Date: 2020-11-24 06:42:30-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc,build/openrpc: remove method example pairings, improve schema examples Removing method example pairings since they were redundant to schema examples and were not implemented well. Improved schema examples by using the ExampleValue method instead of the map lookup. Made a note in the comment here that this is not ideal, since we have to make a shortcut assumption /workaround by using 'unknown' as the method name and the typea as its own parent. Luckily these values aren't heavily used by the method logic. Date: 2020-11-27 12:57:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: use generic number jsonschema for number types Previously used an integer schema assuming hex encoding. It appears, based on review some of the examples, that this may not be the case. Obvioussly this schema could be more descriptive, but just shooting for mostly likely to be not wrong at this point. Date: 2020-12-15 14:44:37-06:00 Signed-off-by: meows <b5c6@protonmail.com> * cmd/lotus,go.mod,go.sum: maybe fix straggling merge resolution conflicts Date: 2021-01-19 12:30:42-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json.gz,build/openrpc/miner.json.gz,build/openrpc/worker.json.gz: run 'make docsgen' Date: 2021-01-19 12:33:55-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/apistruct,node/impl: (lint) gofmt Date: 2021-01-19 12:39:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: maybe fix parse error: open ./api: no such file or directory Date: 2021-01-19 12:52:04-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen,build/openrpc: maybe fix no such file error and run 'make docsgen' Date: 2021-01-19 12:55:52-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: return if AST comment/groupdoc parsing encounters any error This will returns empty comments/docs maps. This should fix issues like: https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218 where the environment makes file lookups hard or impossible. Date: 2021-01-19 13:04:58-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api: Don't depend on build/ * make: support parallel docsgen * openrpc gen: Use simple build version * methodgen * goimports Co-authored-by: meows <b5c6@protonmail.com>
2021-03-19 18:22:46 +00:00
"github.com/filecoin-project/lotus/build"
"github.com/google/uuid"
"github.com/ipfs/go-cid"
"github.com/libp2p/go-libp2p-core/host"
2020-10-22 20:40:26 +00:00
"github.com/libp2p/go-libp2p-core/peer"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-address"
2020-10-01 11:58:26 +00:00
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-fil-markets/piecestore"
2020-07-28 21:35:23 +00:00
retrievalmarket "github.com/filecoin-project/go-fil-markets/retrievalmarket"
storagemarket "github.com/filecoin-project/go-fil-markets/storagemarket"
2020-05-20 18:23:51 +00:00
"github.com/filecoin-project/go-jsonrpc/auth"
2020-09-07 03:49:10 +00:00
"github.com/filecoin-project/go-state-types/abi"
2020-10-01 11:58:26 +00:00
"github.com/filecoin-project/go-state-types/big"
2020-08-17 13:39:33 +00:00
sectorstorage "github.com/filecoin-project/lotus/extern/sector-storage"
"github.com/filecoin-project/lotus/extern/sector-storage/fsutil"
"github.com/filecoin-project/lotus/extern/sector-storage/stores"
"github.com/filecoin-project/lotus/extern/sector-storage/storiface"
2020-08-17 13:39:33 +00:00
sealing "github.com/filecoin-project/lotus/extern/storage-sealing"
2021-06-01 09:56:19 +00:00
"github.com/filecoin-project/lotus/extern/storage-sealing/sealiface"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/apistruct"
OpenRPC Support (#5843) * main: init implement rpc.Discover RPC method This implement the basic functionality for the method over HTTP RPC. Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init example with go-openrpc-reflect lib Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * main: make variable name human-friendly Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init impl of go-openrp-reflect printing document Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * go.mod,go.sum: use go-openrpc-reflect and open-rpc/meta-schema hackforks This is for development only. Versions need to be bumped when they're ready for use as canonical remotes. Signed-off-by: meows <b5c6@protonmail.com> * main,openrpc,main: refactor openrpc supporting code to own package This eliminates code duplication. Signed-off-by: meows <b5c6@protonmail.com> * main: add rpc.Discover to openrpc document Signed-off-by: meows <b5c6@protonmail.com> * openrpc: fix rpc.discover method name casing Also fixes casing stuff for the rest of Filecoin. methods. Signed-off-by: meows <b5c6@protonmail.com> * Revert "main: add rpc.Discover to openrpc document" This reverts commit 116898efb10f33e405ac74acb1aa6daefcd46a62. * main: fix document creation method name This fixes an issue caused with the latest reverting commit. Signed-off-by: meows <b5c6@protonmail.com> * main,docgen,openrpc: refactor to share api parsing, etc as docgen exported stuff Signed-off-by: meows <b5c6@protonmail.com> Makefile: fix docgen refactoring for makefile use of command Signed-off-by: meows <b5c6@protonmail.com> * openrpc: add schema.examples to app reflector There are quite of few of these already registered for the docgen command, so it makes sense to use those! Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init method pairing examples Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump go.mod to use latest meta-schema and openrpc-reflect versions Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init SchemaType mapper function This function will handle the manual configurations for app-specific data types w/r/t their json schema representation. This is useful for cases where the reflect library is unable to provide a sufficient representation automatically. Provided in this commit is an initial implementation for the integerD type (assuming number are represented in the API as hexs), and a commonly used cid.Cid type. Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: tame dependencies by bumping etclabscore/go-openrpc-reflect This removes a problematic dependency on github.com/ethereum/go-ethereum, which was imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect tests. etclabscore/go-openrpc-reflect v0.0.36 has removed this dependency, so this commit is the result of bumping that version and then running 'go mod tidy' This is in response to a review at https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 Date: 2020-11-21 06:52:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: add 'miner' arg to openrpc gen cmd This allows the command to EITHER generate the doc for Full or Miner APIs. See comment for usage. Date: 2020-11-21 07:48:05-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen: add missing examples for Miner API Generating the Miner API OpenRPC doc (via 'go run ./api/openrpc/cmd miner') caused the example logic to panic because some types were missing. This commit adds those missing types, although I'm not an expert in the API so I can't suggest that the example values provided are ideal or well representative. Date: 2020-11-21 07:50:21-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json,build/openrpc/miner.json: add build/openrpc/[full/miner].json docs These will be used as static documents provided by the rpc.discover method. Date: 2020-11-21 07:51:39-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: init go-rice openrpc static assets Date: 2020-11-21 08:23:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: remove rpc.discover implementation from runtime plugin Instead of generating the doc on the fly, we're going to serve a static asset. Rel https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 This removes the runtime implementation from the RPC server construction. Date: 2020-11-21 08:41:20-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,common: add Discover(ctx) method to CommonAPI interface and structs Date: 2020-11-21 08:41:56-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: use rpc server method aliasing for rpc.discover This depends on a currently-forked change at filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210 which establishes this new method RPCServer.AliasMethod. This solves the problem that the OpenRPC spec says that the document should be served at the system extension-prefixed endpoing rpc.discover (not Filecoin.Discover). In fact, the document will be available at BOTH endpoints, but that duplicity is harmless. Date: 2020-11-21 09:18:26-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,build,common: rpc.discover: return json object instead of string Instead of casting the JSON asset from bytes to string, unmarshal it to a map[string]interface{} so the server will provide it as a JSON object. Date: 2020-11-21 09:27:11-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile: merge resolve: docsgen command path Date: 2020-11-22 07:19:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,main,docgen,openrpc: merge resolve: fix func exporteds, signatures Date: 2020-11-22 07:31:03-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: 'get get' auto-bumps version Date: 2020-11-22 07:31:44-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,docgen,main,build/openrpc: refactor openrpc documentation generation This creates Makefile command docsgen-openrpc-json, and refactors the docsgen command to generate both the markdown and openrpc json documents, redirecting the output of the openrpc json documentation to the build/openrpc/ directory, where those json files will be compiled as static assets via go-rice boxes. The api/openrpc/cmd now uses usage argumentation congruent to that of the docgen command (switching on API context). Date: 2020-11-22 08:01:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc: rename api/openrpc -> api/docgen-openrpc Renames the package as well. This is intended to parallel the existing docgen package and command namespacing. Date: 2020-11-22 10:34:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,docgen,build,build/openrpc: use typed Discover response Instead of using a map[string]interface{}, use a typed response for the Discover method implementation. This avoids having to set a docgen Example for the generic map[string]interface{} (as an openrpc document) which both pollutes the generic type and lacks useful information for the Discover method example. Date: 2020-11-22 08:31:16-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,build,main,impl: implement Discover method for Worker and StorageMiner APIs Methods return static compiled assets respective to the APIs. Date: 2020-11-22 08:57:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen_openrpc,build/openrpc: remove timestamping from openrpc doc info This should allow openrpc docs generated at different times to be equal. This is important because the CI (Circle) runs the docgen command and tests that the output and the source are unchanged (via git diff). Date: 2020-11-22 10:47:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc,main,build: fix lint issues Fixes goimports, staticcheck, golint issues. Date: 2020-11-22 11:06:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: fix: don't use an underscore in package name (golint) Date: 2020-11-22 11:07:53-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: fix: mod-tidy-check (run 'go mod tidy') Date: 2020-11-22 11:09:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc dep to latest This version includes the necessary RPCServer.AliasMethod method. Date: 2020-11-23 12:16:15-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,main,build,build/openrpc: init gzipped openrpc static docs Date: 2020-11-24 06:15:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: refactor gzip reading Date: 2020-11-24 06:18:34-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: add basic test for openrpc doc from static assets Date: 2020-11-24 06:30:23-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: handle reader Close error This keeps the errcheck linter happy. Date: 2020-11-24 06:33:14-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: run 'go mod tidy' Date: 2020-11-24 06:36:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: go mod tidy Tidying up after resolving the merge conflicts with master at go.mod Date: 2020-11-24 06:40:45-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc to latest This is a repeat of 76e6fd2, since the latest merge to master seems to have reverted this. Date: 2020-11-24 06:42:30-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc,build/openrpc: remove method example pairings, improve schema examples Removing method example pairings since they were redundant to schema examples and were not implemented well. Improved schema examples by using the ExampleValue method instead of the map lookup. Made a note in the comment here that this is not ideal, since we have to make a shortcut assumption /workaround by using 'unknown' as the method name and the typea as its own parent. Luckily these values aren't heavily used by the method logic. Date: 2020-11-27 12:57:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: use generic number jsonschema for number types Previously used an integer schema assuming hex encoding. It appears, based on review some of the examples, that this may not be the case. Obvioussly this schema could be more descriptive, but just shooting for mostly likely to be not wrong at this point. Date: 2020-12-15 14:44:37-06:00 Signed-off-by: meows <b5c6@protonmail.com> * cmd/lotus,go.mod,go.sum: maybe fix straggling merge resolution conflicts Date: 2021-01-19 12:30:42-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json.gz,build/openrpc/miner.json.gz,build/openrpc/worker.json.gz: run 'make docsgen' Date: 2021-01-19 12:33:55-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/apistruct,node/impl: (lint) gofmt Date: 2021-01-19 12:39:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: maybe fix parse error: open ./api: no such file or directory Date: 2021-01-19 12:52:04-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen,build/openrpc: maybe fix no such file error and run 'make docsgen' Date: 2021-01-19 12:55:52-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: return if AST comment/groupdoc parsing encounters any error This will returns empty comments/docs maps. This should fix issues like: https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218 where the environment makes file lookups hard or impossible. Date: 2021-01-19 13:04:58-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api: Don't depend on build/ * make: support parallel docsgen * openrpc gen: Use simple build version * methodgen * goimports Co-authored-by: meows <b5c6@protonmail.com>
2021-03-19 18:22:46 +00:00
apitypes "github.com/filecoin-project/lotus/api/types"
"github.com/filecoin-project/lotus/chain/types"
2021-02-05 17:58:55 +00:00
"github.com/filecoin-project/lotus/markets/storageadapter"
2019-11-25 04:45:13 +00:00
"github.com/filecoin-project/lotus/miner"
"github.com/filecoin-project/lotus/node/impl/common"
"github.com/filecoin-project/lotus/node/modules/dtypes"
"github.com/filecoin-project/lotus/storage"
"github.com/filecoin-project/lotus/storage/sectorblocks"
2020-11-26 07:02:43 +00:00
sto "github.com/filecoin-project/specs-storage/storage"
2019-07-24 00:58:31 +00:00
)
type StorageMinerAPI struct {
common.CommonAPI
2019-07-27 00:45:27 +00:00
2020-03-05 19:21:06 +00:00
SectorBlocks *sectorblocks.SectorBlocks
2020-07-28 21:35:23 +00:00
PieceStore dtypes.ProviderPieceStore
StorageProvider storagemarket.StorageProvider
RetrievalProvider retrievalmarket.RetrievalProvider
Miner *storage.Miner
BlockMiner *miner.Miner
Full api.FullNode
StorageMgr *sectorstorage.Manager `optional:"true"`
IStorageMgr sectorstorage.SectorManager
*stores.Index
2020-09-07 14:12:46 +00:00
storiface.WorkerReturn
2021-02-05 17:58:55 +00:00
DataTransfer dtypes.ProviderDataTransfer
Host host.Host
AddrSel *storage.AddressSelector
DealPublisher *storageadapter.DealPublisher
2021-03-26 05:32:03 +00:00
Epp gen.WinningPoStProver
DS dtypes.MetadataDS
2020-10-01 11:58:26 +00:00
ConsiderOnlineStorageDealsConfigFunc dtypes.ConsiderOnlineStorageDealsConfigFunc
SetConsiderOnlineStorageDealsConfigFunc dtypes.SetConsiderOnlineStorageDealsConfigFunc
ConsiderOnlineRetrievalDealsConfigFunc dtypes.ConsiderOnlineRetrievalDealsConfigFunc
SetConsiderOnlineRetrievalDealsConfigFunc dtypes.SetConsiderOnlineRetrievalDealsConfigFunc
StorageDealPieceCidBlocklistConfigFunc dtypes.StorageDealPieceCidBlocklistConfigFunc
SetStorageDealPieceCidBlocklistConfigFunc dtypes.SetStorageDealPieceCidBlocklistConfigFunc
ConsiderOfflineStorageDealsConfigFunc dtypes.ConsiderOfflineStorageDealsConfigFunc
SetConsiderOfflineStorageDealsConfigFunc dtypes.SetConsiderOfflineStorageDealsConfigFunc
ConsiderOfflineRetrievalDealsConfigFunc dtypes.ConsiderOfflineRetrievalDealsConfigFunc
SetConsiderOfflineRetrievalDealsConfigFunc dtypes.SetConsiderOfflineRetrievalDealsConfigFunc
ConsiderVerifiedStorageDealsConfigFunc dtypes.ConsiderVerifiedStorageDealsConfigFunc
SetConsiderVerifiedStorageDealsConfigFunc dtypes.SetConsiderVerifiedStorageDealsConfigFunc
ConsiderUnverifiedStorageDealsConfigFunc dtypes.ConsiderUnverifiedStorageDealsConfigFunc
SetConsiderUnverifiedStorageDealsConfigFunc dtypes.SetConsiderUnverifiedStorageDealsConfigFunc
SetSealingConfigFunc dtypes.SetSealingConfigFunc
GetSealingConfigFunc dtypes.GetSealingConfigFunc
GetExpectedSealDurationFunc dtypes.GetExpectedSealDurationFunc
SetExpectedSealDurationFunc dtypes.SetExpectedSealDurationFunc
2019-07-24 00:58:31 +00:00
}
func (sm *StorageMinerAPI) ServeRemote(w http.ResponseWriter, r *http.Request) {
2020-05-20 18:23:51 +00:00
if !auth.HasPerm(r.Context(), nil, apistruct.PermAdmin) {
w.WriteHeader(401)
_ = json.NewEncoder(w).Encode(struct{ Error string }{"unauthorized: missing write permission"})
return
}
2020-03-11 05:49:17 +00:00
sm.StorageMgr.ServeHTTP(w, r)
}
2020-03-05 19:21:06 +00:00
func (sm *StorageMinerAPI) WorkerStats(context.Context) (map[uuid.UUID]storiface.WorkerStats, error) {
2020-03-23 14:56:22 +00:00
return sm.StorageMgr.WorkerStats(), nil
}
2019-11-08 18:15:13 +00:00
func (sm *StorageMinerAPI) WorkerJobs(ctx context.Context) (map[uuid.UUID][]storiface.WorkerJob, error) {
2020-07-21 18:07:49 +00:00
return sm.StorageMgr.WorkerJobs(), nil
}
func (sm *StorageMinerAPI) ActorAddress(context.Context) (address.Address, error) {
2020-03-17 20:19:52 +00:00
return sm.Miner.Address(), nil
2019-08-10 01:54:45 +00:00
}
2020-04-23 21:12:42 +00:00
func (sm *StorageMinerAPI) MiningBase(ctx context.Context) (*types.TipSet, error) {
mb, err := sm.BlockMiner.GetBestMiningCandidate(ctx)
if err != nil {
return nil, err
}
return mb.TipSet, nil
}
func (sm *StorageMinerAPI) ActorSectorSize(ctx context.Context, addr address.Address) (abi.SectorSize, error) {
mi, err := sm.Full.StateMinerInfo(ctx, addr, types.EmptyTSK)
if err != nil {
return 0, err
}
return mi.SectorSize, nil
}
2021-02-16 18:16:35 +00:00
func (sm *StorageMinerAPI) PledgeSector(ctx context.Context) (abi.SectorID, error) {
sr, err := sm.Miner.PledgeSector(ctx)
if err != nil {
2021-02-16 18:16:35 +00:00
return abi.SectorID{}, err
}
// wait for the sector to enter the Packing state
// TODO: instead of polling implement some pubsub-type thing in storagefsm
for {
info, err := sm.Miner.GetSectorInfo(sr.ID.Number)
if err != nil {
2021-02-16 18:16:35 +00:00
return abi.SectorID{}, xerrors.Errorf("getting pledged sector info: %w", err)
}
if info.State != sealing.UndefinedSectorState {
2021-02-16 18:16:35 +00:00
return sr.ID, nil
}
select {
case <-time.After(10 * time.Millisecond):
case <-ctx.Done():
2021-02-16 18:16:35 +00:00
return abi.SectorID{}, ctx.Err()
}
}
2019-07-27 01:54:03 +00:00
}
func (sm *StorageMinerAPI) SectorsStatus(ctx context.Context, sid abi.SectorNumber, showOnChainInfo bool) (api.SectorInfo, error) {
2019-11-08 18:15:13 +00:00
info, err := sm.Miner.GetSectorInfo(sid)
if err != nil {
return api.SectorInfo{}, err
}
deals := make([]abi.DealID, len(info.Pieces))
2019-11-08 18:15:13 +00:00
for i, piece := range info.Pieces {
if piece.DealInfo == nil {
2020-02-23 00:47:47 +00:00
continue
}
deals[i] = piece.DealInfo.DealID
2019-11-08 18:15:13 +00:00
}
log := make([]api.SectorLog, len(info.Log))
for i, l := range info.Log {
log[i] = api.SectorLog{
Kind: l.Kind,
Timestamp: l.Timestamp,
Trace: l.Trace,
Message: l.Message,
}
}
sInfo := api.SectorInfo{
2019-11-08 18:15:13 +00:00
SectorID: sid,
State: api.SectorState(info.State),
2019-11-08 18:15:13 +00:00
CommD: info.CommD,
CommR: info.CommR,
Proof: info.Proof,
Deals: deals,
Ticket: api.SealTicket{
Value: info.TicketValue,
Epoch: info.TicketEpoch,
},
Seed: api.SealSeed{
Value: info.SeedValue,
Epoch: info.SeedEpoch,
},
PreCommitMsg: info.PreCommitMessage,
2020-08-27 10:41:24 +00:00
CommitMsg: info.CommitMessage,
Retries: info.InvalidProofs,
ToUpgrade: sm.Miner.IsMarkedForUpgrade(sid),
2019-12-04 00:44:29 +00:00
LastErr: info.LastErr,
Log: log,
// on chain info
2020-07-27 23:22:41 +00:00
SealProof: 0,
Activation: 0,
Expiration: 0,
DealWeight: big.Zero(),
VerifiedDealWeight: big.Zero(),
2020-07-27 23:22:41 +00:00
InitialPledge: big.Zero(),
OnTime: 0,
Early: 0,
}
if !showOnChainInfo {
return sInfo, nil
}
onChainInfo, err := sm.Full.StateSectorGetInfo(ctx, sm.Miner.Address(), sid, types.EmptyTSK)
if err != nil {
return sInfo, err
}
if onChainInfo == nil {
return sInfo, nil
}
sInfo.SealProof = onChainInfo.SealProof
sInfo.Activation = onChainInfo.Activation
sInfo.Expiration = onChainInfo.Expiration
sInfo.DealWeight = onChainInfo.DealWeight
sInfo.VerifiedDealWeight = onChainInfo.VerifiedDealWeight
sInfo.InitialPledge = onChainInfo.InitialPledge
ex, err := sm.Full.StateSectorExpiration(ctx, sm.Miner.Address(), sid, types.EmptyTSK)
if err != nil {
return sInfo, nil
}
sInfo.OnTime = ex.OnTime
sInfo.Early = ex.Early
return sInfo, nil
}
// List all staged sectors
func (sm *StorageMinerAPI) SectorsList(context.Context) ([]abi.SectorNumber, error) {
2019-11-08 18:15:13 +00:00
sectors, err := sm.Miner.ListSectors()
if err != nil {
return nil, err
}
out := make([]abi.SectorNumber, len(sectors))
2019-11-08 18:15:13 +00:00
for i, sector := range sectors {
if sector.State == sealing.UndefinedSectorState {
continue // sector ID not set yet
}
2020-04-06 22:31:33 +00:00
out[i] = sector.SectorNumber
2019-11-08 18:15:13 +00:00
}
return out, nil
}
2020-12-06 00:51:48 +00:00
func (sm *StorageMinerAPI) SectorsListInStates(ctx context.Context, states []api.SectorState) ([]abi.SectorNumber, error) {
filterStates := make(map[sealing.SectorState]struct{})
for _, state := range states {
st := sealing.SectorState(state)
if _, ok := sealing.ExistSectorStateList[st]; !ok {
continue
}
filterStates[st] = struct{}{}
}
var sns []abi.SectorNumber
if len(filterStates) == 0 {
return sns, nil
}
sectors, err := sm.Miner.ListSectors()
if err != nil {
return nil, err
}
for i := range sectors {
if _, ok := filterStates[sectors[i].State]; ok {
sns = append(sns, sectors[i].SectorNumber)
}
}
return sns, nil
}
func (sm *StorageMinerAPI) SectorsSummary(ctx context.Context) (map[api.SectorState]int, error) {
sectors, err := sm.Miner.ListSectors()
if err != nil {
return nil, err
}
out := make(map[api.SectorState]int)
for i := range sectors {
state := api.SectorState(sectors[i].State)
2020-12-10 18:36:02 +00:00
out[state]++
2020-12-06 00:51:48 +00:00
}
return out, nil
}
2020-03-19 19:51:33 +00:00
func (sm *StorageMinerAPI) StorageLocal(ctx context.Context) (map[stores.ID]string, error) {
return sm.StorageMgr.StorageLocal(ctx)
}
2019-08-26 10:04:57 +00:00
func (sm *StorageMinerAPI) SectorsRefs(context.Context) (map[string][]api.SealedRef, error) {
// json can't handle cids as map keys
out := map[string][]api.SealedRef{}
refs, err := sm.SectorBlocks.List()
if err != nil {
return nil, err
}
for k, v := range refs {
out[strconv.FormatUint(k, 10)] = v
2019-08-26 10:04:57 +00:00
}
return out, nil
}
2020-07-08 15:23:27 +00:00
func (sm *StorageMinerAPI) StorageStat(ctx context.Context, id stores.ID) (fsutil.FsStat, error) {
return sm.StorageMgr.FsStat(ctx, id)
}
2020-06-26 15:28:05 +00:00
func (sm *StorageMinerAPI) SectorStartSealing(ctx context.Context, number abi.SectorNumber) error {
return sm.Miner.StartPackingSector(number)
}
func (sm *StorageMinerAPI) SectorSetSealDelay(ctx context.Context, delay time.Duration) error {
2020-08-18 14:20:31 +00:00
cfg, err := sm.GetSealingConfigFunc()
if err != nil {
return xerrors.Errorf("get config: %w", err)
}
cfg.WaitDealsDelay = delay
return sm.SetSealingConfigFunc(cfg)
}
func (sm *StorageMinerAPI) SectorGetSealDelay(ctx context.Context) (time.Duration, error) {
2020-08-18 14:20:31 +00:00
cfg, err := sm.GetSealingConfigFunc()
if err != nil {
return 0, err
}
return cfg.WaitDealsDelay, nil
}
func (sm *StorageMinerAPI) SectorSetExpectedSealDuration(ctx context.Context, delay time.Duration) error {
return sm.SetExpectedSealDurationFunc(delay)
}
func (sm *StorageMinerAPI) SectorGetExpectedSealDuration(ctx context.Context) (time.Duration, error) {
return sm.GetExpectedSealDurationFunc()
}
func (sm *StorageMinerAPI) SectorsUpdate(ctx context.Context, id abi.SectorNumber, state api.SectorState) error {
return sm.Miner.ForceSectorState(ctx, id, sealing.SectorState(state))
}
2020-03-05 19:21:06 +00:00
2020-06-22 17:35:14 +00:00
func (sm *StorageMinerAPI) SectorRemove(ctx context.Context, id abi.SectorNumber) error {
return sm.Miner.RemoveSector(ctx, id)
}
2021-01-12 23:42:01 +00:00
func (sm *StorageMinerAPI) SectorTerminate(ctx context.Context, id abi.SectorNumber) error {
return sm.Miner.TerminateSector(ctx, id)
}
2021-01-13 22:32:04 +00:00
func (sm *StorageMinerAPI) SectorTerminateFlush(ctx context.Context) (*cid.Cid, error) {
return sm.Miner.TerminateFlush(ctx)
}
2021-01-14 11:37:23 +00:00
func (sm *StorageMinerAPI) SectorTerminatePending(ctx context.Context) ([]abi.SectorID, error) {
return sm.Miner.TerminatePending(ctx)
}
func (sm *StorageMinerAPI) SectorPreCommitFlush(ctx context.Context) ([]sealiface.PreCommitBatchRes, error) {
2021-05-18 14:51:06 +00:00
return sm.Miner.SectorPreCommitFlush(ctx)
}
func (sm *StorageMinerAPI) SectorPreCommitPending(ctx context.Context) ([]abi.SectorID, error) {
return sm.Miner.SectorPreCommitPending(ctx)
}
2020-07-01 14:49:17 +00:00
func (sm *StorageMinerAPI) SectorMarkForUpgrade(ctx context.Context, id abi.SectorNumber) error {
return sm.Miner.MarkForUpgrade(id)
}
2021-06-01 09:56:19 +00:00
func (sm *StorageMinerAPI) SectorCommitFlush(ctx context.Context) ([]sealiface.CommitBatchRes, error) {
2021-03-10 15:16:44 +00:00
return sm.Miner.CommitFlush(ctx)
}
func (sm *StorageMinerAPI) SectorCommitPending(ctx context.Context) ([]abi.SectorID, error) {
return sm.Miner.CommitPending(ctx)
}
2020-03-11 01:57:52 +00:00
func (sm *StorageMinerAPI) WorkerConnect(ctx context.Context, url string) error {
w, err := connectRemoteWorker(ctx, sm, url)
2020-03-11 01:57:52 +00:00
if err != nil {
2020-03-18 23:23:28 +00:00
return xerrors.Errorf("connecting remote storage failed: %w", err)
2020-03-11 01:57:52 +00:00
}
2020-03-18 23:23:28 +00:00
log.Infof("Connected to a remote worker at %s", url)
2020-03-20 22:30:17 +00:00
return sm.StorageMgr.AddWorker(ctx, w)
2020-03-11 01:57:52 +00:00
}
func (sm *StorageMinerAPI) SealingSchedDiag(ctx context.Context, doSched bool) (interface{}, error) {
return sm.StorageMgr.SchedDiag(ctx, doSched)
}
func (sm *StorageMinerAPI) SealingAbort(ctx context.Context, call storiface.CallID) error {
return sm.StorageMgr.Abort(ctx, call)
}
func (sm *StorageMinerAPI) MarketImportDealData(ctx context.Context, propCid cid.Cid, path string) error {
fi, err := os.Open(path)
if err != nil {
return xerrors.Errorf("failed to open file: %w", err)
}
defer fi.Close() //nolint:errcheck
return sm.StorageProvider.ImportDataForDeal(ctx, propCid, fi)
}
func (sm *StorageMinerAPI) listDeals(ctx context.Context) ([]api.MarketDeal, error) {
ts, err := sm.Full.ChainHead(ctx)
if err != nil {
return nil, err
}
tsk := ts.Key()
allDeals, err := sm.Full.StateMarketDeals(ctx, tsk)
if err != nil {
return nil, err
}
var out []api.MarketDeal
for _, deal := range allDeals {
if deal.Proposal.Provider == sm.Miner.Address() {
out = append(out, deal)
}
}
return out, nil
}
func (sm *StorageMinerAPI) MarketListDeals(ctx context.Context) ([]api.MarketDeal, error) {
2020-09-17 08:17:14 +00:00
return sm.listDeals(ctx)
}
2020-08-04 23:40:29 +00:00
func (sm *StorageMinerAPI) MarketListRetrievalDeals(ctx context.Context) ([]retrievalmarket.ProviderDealState, error) {
var out []retrievalmarket.ProviderDealState
deals := sm.RetrievalProvider.ListDeals()
for _, deal := range deals {
out = append(out, deal)
}
return out, nil
}
func (sm *StorageMinerAPI) MarketGetDealUpdates(ctx context.Context) (<-chan storagemarket.MinerDeal, error) {
2020-08-06 20:16:55 +00:00
results := make(chan storagemarket.MinerDeal)
unsub := sm.StorageProvider.SubscribeToEvents(func(evt storagemarket.ProviderEvent, deal storagemarket.MinerDeal) {
select {
case results <- deal:
case <-ctx.Done():
2020-08-06 20:16:55 +00:00
}
})
go func() {
<-ctx.Done()
unsub()
close(results)
}()
return results, nil
}
func (sm *StorageMinerAPI) MarketListIncompleteDeals(ctx context.Context) ([]storagemarket.MinerDeal, error) {
return sm.StorageProvider.ListLocalDeals()
}
2020-07-31 19:14:48 +00:00
func (sm *StorageMinerAPI) MarketSetAsk(ctx context.Context, price types.BigInt, verifiedPrice types.BigInt, duration abi.ChainEpoch, minPieceSize abi.PaddedPieceSize, maxPieceSize abi.PaddedPieceSize) error {
options := []storagemarket.StorageAskOption{
storagemarket.MinPieceSize(minPieceSize),
storagemarket.MaxPieceSize(maxPieceSize),
}
2020-07-31 19:14:48 +00:00
return sm.StorageProvider.SetAsk(price, verifiedPrice, duration, options...)
}
2020-06-17 00:18:54 +00:00
func (sm *StorageMinerAPI) MarketGetAsk(ctx context.Context) (*storagemarket.SignedStorageAsk, error) {
return sm.StorageProvider.GetAsk(), nil
}
2020-07-28 21:35:23 +00:00
func (sm *StorageMinerAPI) MarketSetRetrievalAsk(ctx context.Context, rask *retrievalmarket.Ask) error {
sm.RetrievalProvider.SetAsk(rask)
return nil
}
func (sm *StorageMinerAPI) MarketGetRetrievalAsk(ctx context.Context) (*retrievalmarket.Ask, error) {
return sm.RetrievalProvider.GetAsk(), nil
}
func (sm *StorageMinerAPI) MarketListDataTransfers(ctx context.Context) ([]api.DataTransferChannel, error) {
inProgressChannels, err := sm.DataTransfer.InProgressChannels(ctx)
if err != nil {
return nil, err
}
apiChannels := make([]api.DataTransferChannel, 0, len(inProgressChannels))
for _, channelState := range inProgressChannels {
apiChannels = append(apiChannels, api.NewDataTransferChannel(sm.Host.ID(), channelState))
}
return apiChannels, nil
}
2020-10-22 20:40:26 +00:00
func (sm *StorageMinerAPI) MarketRestartDataTransfer(ctx context.Context, transferID datatransfer.TransferID, otherPeer peer.ID, isInitiator bool) error {
selfPeer := sm.Host.ID()
if isInitiator {
return sm.DataTransfer.RestartDataTransferChannel(ctx, datatransfer.ChannelID{Initiator: selfPeer, Responder: otherPeer, ID: transferID})
}
return sm.DataTransfer.RestartDataTransferChannel(ctx, datatransfer.ChannelID{Initiator: otherPeer, Responder: selfPeer, ID: transferID})
}
func (sm *StorageMinerAPI) MarketCancelDataTransfer(ctx context.Context, transferID datatransfer.TransferID, otherPeer peer.ID, isInitiator bool) error {
selfPeer := sm.Host.ID()
if isInitiator {
return sm.DataTransfer.CloseDataTransferChannel(ctx, datatransfer.ChannelID{Initiator: selfPeer, Responder: otherPeer, ID: transferID})
}
return sm.DataTransfer.CloseDataTransferChannel(ctx, datatransfer.ChannelID{Initiator: otherPeer, Responder: selfPeer, ID: transferID})
}
func (sm *StorageMinerAPI) MarketDataTransferUpdates(ctx context.Context) (<-chan api.DataTransferChannel, error) {
channels := make(chan api.DataTransferChannel)
unsub := sm.DataTransfer.SubscribeToEvents(func(evt datatransfer.Event, channelState datatransfer.ChannelState) {
channel := api.NewDataTransferChannel(sm.Host.ID(), channelState)
select {
case <-ctx.Done():
case channels <- channel:
}
})
go func() {
defer unsub()
<-ctx.Done()
}()
return channels, nil
}
2021-02-05 17:58:55 +00:00
func (sm *StorageMinerAPI) MarketPendingDeals(ctx context.Context) (api.PendingDealInfo, error) {
return sm.DealPublisher.PendingDeals(), nil
}
func (sm *StorageMinerAPI) MarketPublishPendingDeals(ctx context.Context) error {
sm.DealPublisher.ForcePublishPendingDeals()
return nil
}
func (sm *StorageMinerAPI) DealsList(ctx context.Context) ([]api.MarketDeal, error) {
return sm.listDeals(ctx)
}
2020-08-04 23:40:29 +00:00
func (sm *StorageMinerAPI) RetrievalDealsList(ctx context.Context) (map[retrievalmarket.ProviderDealIdentifier]retrievalmarket.ProviderDealState, error) {
return sm.RetrievalProvider.ListDeals(), nil
}
func (sm *StorageMinerAPI) DealsConsiderOnlineStorageDeals(ctx context.Context) (bool, error) {
return sm.ConsiderOnlineStorageDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderOnlineStorageDeals(ctx context.Context, b bool) error {
return sm.SetConsiderOnlineStorageDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsConsiderOnlineRetrievalDeals(ctx context.Context) (bool, error) {
return sm.ConsiderOnlineRetrievalDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderOnlineRetrievalDeals(ctx context.Context, b bool) error {
return sm.SetConsiderOnlineRetrievalDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsConsiderOfflineStorageDeals(ctx context.Context) (bool, error) {
return sm.ConsiderOfflineStorageDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderOfflineStorageDeals(ctx context.Context, b bool) error {
return sm.SetConsiderOfflineStorageDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsConsiderOfflineRetrievalDeals(ctx context.Context) (bool, error) {
return sm.ConsiderOfflineRetrievalDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderOfflineRetrievalDeals(ctx context.Context, b bool) error {
return sm.SetConsiderOfflineRetrievalDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsConsiderVerifiedStorageDeals(ctx context.Context) (bool, error) {
return sm.ConsiderVerifiedStorageDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderVerifiedStorageDeals(ctx context.Context, b bool) error {
return sm.SetConsiderVerifiedStorageDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsConsiderUnverifiedStorageDeals(ctx context.Context) (bool, error) {
return sm.ConsiderUnverifiedStorageDealsConfigFunc()
}
func (sm *StorageMinerAPI) DealsSetConsiderUnverifiedStorageDeals(ctx context.Context, b bool) error {
return sm.SetConsiderUnverifiedStorageDealsConfigFunc(b)
}
func (sm *StorageMinerAPI) DealsGetExpectedSealDurationFunc(ctx context.Context) (time.Duration, error) {
return sm.GetExpectedSealDurationFunc()
}
func (sm *StorageMinerAPI) DealsSetExpectedSealDurationFunc(ctx context.Context, d time.Duration) error {
return sm.SetExpectedSealDurationFunc(d)
}
func (sm *StorageMinerAPI) DealsImportData(ctx context.Context, deal cid.Cid, fname string) error {
fi, err := os.Open(fname)
if err != nil {
return xerrors.Errorf("failed to open given file: %w", err)
}
defer fi.Close() //nolint:errcheck
return sm.StorageProvider.ImportDataForDeal(ctx, deal, fi)
}
2020-06-18 22:42:24 +00:00
func (sm *StorageMinerAPI) DealsPieceCidBlocklist(ctx context.Context) ([]cid.Cid, error) {
return sm.StorageDealPieceCidBlocklistConfigFunc()
}
2020-06-18 22:42:24 +00:00
func (sm *StorageMinerAPI) DealsSetPieceCidBlocklist(ctx context.Context, cids []cid.Cid) error {
return sm.SetStorageDealPieceCidBlocklistConfigFunc(cids)
}
func (sm *StorageMinerAPI) StorageAddLocal(ctx context.Context, path string) error {
if sm.StorageMgr == nil {
return xerrors.Errorf("no storage manager")
}
2020-03-19 15:10:19 +00:00
return sm.StorageMgr.AddLocalStorage(ctx, path)
}
func (sm *StorageMinerAPI) PiecesListPieces(ctx context.Context) ([]cid.Cid, error) {
2020-07-28 21:35:23 +00:00
return sm.PieceStore.ListPieceInfoKeys()
}
func (sm *StorageMinerAPI) PiecesListCidInfos(ctx context.Context) ([]cid.Cid, error) {
2020-07-28 21:35:23 +00:00
return sm.PieceStore.ListCidInfoKeys()
}
func (sm *StorageMinerAPI) PiecesGetPieceInfo(ctx context.Context, pieceCid cid.Cid) (*piecestore.PieceInfo, error) {
pi, err := sm.PieceStore.GetPieceInfo(pieceCid)
if err != nil {
return nil, err
}
return &pi, nil
}
func (sm *StorageMinerAPI) PiecesGetCIDInfo(ctx context.Context, payloadCid cid.Cid) (*piecestore.CIDInfo, error) {
ci, err := sm.PieceStore.GetCIDInfo(payloadCid)
if err != nil {
return nil, err
}
return &ci, nil
}
2020-10-01 11:58:26 +00:00
func (sm *StorageMinerAPI) CreateBackup(ctx context.Context, fpath string) error {
2020-10-01 15:14:08 +00:00
return backup(sm.DS, fpath)
2020-10-01 11:58:26 +00:00
}
2020-12-01 23:32:01 +00:00
func (sm *StorageMinerAPI) CheckProvable(ctx context.Context, pp abi.RegisteredPoStProof, sectors []sto.SectorRef, expensive bool) (map[abi.SectorNumber]string, error) {
2020-12-01 23:35:55 +00:00
var rg storiface.RGetter
2020-12-01 23:32:01 +00:00
if expensive {
rg = func(ctx context.Context, id abi.SectorID) (cid.Cid, error) {
si, err := sm.Miner.GetSectorInfo(id.Number)
if err != nil {
return cid.Undef, err
}
if si.CommR == nil {
return cid.Undef, xerrors.Errorf("commr is nil")
}
return *si.CommR, nil
}
}
bad, err := sm.StorageMgr.CheckProvable(ctx, pp, sectors, rg)
2020-11-26 07:02:43 +00:00
if err != nil {
return nil, err
}
var out = make(map[abi.SectorNumber]string)
for sid, err := range bad {
out[sid.Number] = err
}
return out, nil
}
2020-12-02 19:46:07 +00:00
func (sm *StorageMinerAPI) ActorAddressConfig(ctx context.Context) (api.AddressConfig, error) {
return sm.AddrSel.AddressConfig, nil
}
OpenRPC Support (#5843) * main: init implement rpc.Discover RPC method This implement the basic functionality for the method over HTTP RPC. Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init example with go-openrpc-reflect lib Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * main: make variable name human-friendly Signed-off-by: meows <b5c6@protonmail.com> * main,go.mod,go.sum: init impl of go-openrp-reflect printing document Signed-off-by: meows <b5c6@protonmail.com> Conflicts: go.mod go.sum * go.mod,go.sum: use go-openrpc-reflect and open-rpc/meta-schema hackforks This is for development only. Versions need to be bumped when they're ready for use as canonical remotes. Signed-off-by: meows <b5c6@protonmail.com> * main,openrpc,main: refactor openrpc supporting code to own package This eliminates code duplication. Signed-off-by: meows <b5c6@protonmail.com> * main: add rpc.Discover to openrpc document Signed-off-by: meows <b5c6@protonmail.com> * openrpc: fix rpc.discover method name casing Also fixes casing stuff for the rest of Filecoin. methods. Signed-off-by: meows <b5c6@protonmail.com> * Revert "main: add rpc.Discover to openrpc document" This reverts commit 116898efb10f33e405ac74acb1aa6daefcd46a62. * main: fix document creation method name This fixes an issue caused with the latest reverting commit. Signed-off-by: meows <b5c6@protonmail.com> * main,docgen,openrpc: refactor to share api parsing, etc as docgen exported stuff Signed-off-by: meows <b5c6@protonmail.com> Makefile: fix docgen refactoring for makefile use of command Signed-off-by: meows <b5c6@protonmail.com> * openrpc: add schema.examples to app reflector There are quite of few of these already registered for the docgen command, so it makes sense to use those! Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init method pairing examples Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump go.mod to use latest meta-schema and openrpc-reflect versions Signed-off-by: meows <b5c6@protonmail.com> * openrpc: init SchemaType mapper function This function will handle the manual configurations for app-specific data types w/r/t their json schema representation. This is useful for cases where the reflect library is unable to provide a sufficient representation automatically. Provided in this commit is an initial implementation for the integerD type (assuming number are represented in the API as hexs), and a commonly used cid.Cid type. Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: tame dependencies by bumping etclabscore/go-openrpc-reflect This removes a problematic dependency on github.com/ethereum/go-ethereum, which was imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect tests. etclabscore/go-openrpc-reflect v0.0.36 has removed this dependency, so this commit is the result of bumping that version and then running 'go mod tidy' This is in response to a review at https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 Date: 2020-11-21 06:52:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: add 'miner' arg to openrpc gen cmd This allows the command to EITHER generate the doc for Full or Miner APIs. See comment for usage. Date: 2020-11-21 07:48:05-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen: add missing examples for Miner API Generating the Miner API OpenRPC doc (via 'go run ./api/openrpc/cmd miner') caused the example logic to panic because some types were missing. This commit adds those missing types, although I'm not an expert in the API so I can't suggest that the example values provided are ideal or well representative. Date: 2020-11-21 07:50:21-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json,build/openrpc/miner.json: add build/openrpc/[full/miner].json docs These will be used as static documents provided by the rpc.discover method. Date: 2020-11-21 07:51:39-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: init go-rice openrpc static assets Date: 2020-11-21 08:23:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: remove rpc.discover implementation from runtime plugin Instead of generating the doc on the fly, we're going to serve a static asset. Rel https://github.com/filecoin-project/lotus/pull/4711#pullrequestreview-535686205 This removes the runtime implementation from the RPC server construction. Date: 2020-11-21 08:41:20-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,common: add Discover(ctx) method to CommonAPI interface and structs Date: 2020-11-21 08:41:56-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main: use rpc server method aliasing for rpc.discover This depends on a currently-forked change at filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210 which establishes this new method RPCServer.AliasMethod. This solves the problem that the OpenRPC spec says that the document should be served at the system extension-prefixed endpoing rpc.discover (not Filecoin.Discover). In fact, the document will be available at BOTH endpoints, but that duplicity is harmless. Date: 2020-11-21 09:18:26-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,build,common: rpc.discover: return json object instead of string Instead of casting the JSON asset from bytes to string, unmarshal it to a map[string]interface{} so the server will provide it as a JSON object. Date: 2020-11-21 09:27:11-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile: merge resolve: docsgen command path Date: 2020-11-22 07:19:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,main,docgen,openrpc: merge resolve: fix func exporteds, signatures Date: 2020-11-22 07:31:03-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: 'get get' auto-bumps version Date: 2020-11-22 07:31:44-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,docgen,main,build/openrpc: refactor openrpc documentation generation This creates Makefile command docsgen-openrpc-json, and refactors the docsgen command to generate both the markdown and openrpc json documents, redirecting the output of the openrpc json documentation to the build/openrpc/ directory, where those json files will be compiled as static assets via go-rice boxes. The api/openrpc/cmd now uses usage argumentation congruent to that of the docgen command (switching on API context). Date: 2020-11-22 08:01:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc: rename api/openrpc -> api/docgen-openrpc Renames the package as well. This is intended to parallel the existing docgen package and command namespacing. Date: 2020-11-22 10:34:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api,apistruct,docgen,build,build/openrpc: use typed Discover response Instead of using a map[string]interface{}, use a typed response for the Discover method implementation. This avoids having to set a docgen Example for the generic map[string]interface{} (as an openrpc document) which both pollutes the generic type and lacks useful information for the Discover method example. Date: 2020-11-22 08:31:16-06:00 Signed-off-by: meows <b5c6@protonmail.com> * apistruct,build,main,impl: implement Discover method for Worker and StorageMiner APIs Methods return static compiled assets respective to the APIs. Date: 2020-11-22 08:57:18-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgen_openrpc,build/openrpc: remove timestamping from openrpc doc info This should allow openrpc docs generated at different times to be equal. This is important because the CI (Circle) runs the docgen command and tests that the output and the source are unchanged (via git diff). Date: 2020-11-22 10:47:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * main,docgen_openrpc,main,build: fix lint issues Fixes goimports, staticcheck, golint issues. Date: 2020-11-22 11:06:46-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: fix: don't use an underscore in package name (golint) Date: 2020-11-22 11:07:53-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: fix: mod-tidy-check (run 'go mod tidy') Date: 2020-11-22 11:09:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc dep to latest This version includes the necessary RPCServer.AliasMethod method. Date: 2020-11-23 12:16:15-06:00 Signed-off-by: meows <b5c6@protonmail.com> * Makefile,main,build,build/openrpc: init gzipped openrpc static docs Date: 2020-11-24 06:15:06-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: refactor gzip reading Date: 2020-11-24 06:18:34-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: add basic test for openrpc doc from static assets Date: 2020-11-24 06:30:23-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build: handle reader Close error This keeps the errcheck linter happy. Date: 2020-11-24 06:33:14-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.sum: run 'go mod tidy' Date: 2020-11-24 06:36:07-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: go mod tidy Tidying up after resolving the merge conflicts with master at go.mod Date: 2020-11-24 06:40:45-06:00 Signed-off-by: meows <b5c6@protonmail.com> * go.mod,go.sum: bump filecoin-project/go-jsonrpc to latest This is a repeat of 76e6fd2, since the latest merge to master seems to have reverted this. Date: 2020-11-24 06:42:30-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc,build/openrpc: remove method example pairings, improve schema examples Removing method example pairings since they were redundant to schema examples and were not implemented well. Improved schema examples by using the ExampleValue method instead of the map lookup. Made a note in the comment here that this is not ideal, since we have to make a shortcut assumption /workaround by using 'unknown' as the method name and the typea as its own parent. Luckily these values aren't heavily used by the method logic. Date: 2020-11-27 12:57:36-06:00 Signed-off-by: meows <b5c6@protonmail.com> * docgenopenrpc: use generic number jsonschema for number types Previously used an integer schema assuming hex encoding. It appears, based on review some of the examples, that this may not be the case. Obvioussly this schema could be more descriptive, but just shooting for mostly likely to be not wrong at this point. Date: 2020-12-15 14:44:37-06:00 Signed-off-by: meows <b5c6@protonmail.com> * cmd/lotus,go.mod,go.sum: maybe fix straggling merge resolution conflicts Date: 2021-01-19 12:30:42-06:00 Signed-off-by: meows <b5c6@protonmail.com> * build/openrpc/full.json.gz,build/openrpc/miner.json.gz,build/openrpc/worker.json.gz: run 'make docsgen' Date: 2021-01-19 12:33:55-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/apistruct,node/impl: (lint) gofmt Date: 2021-01-19 12:39:48-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: maybe fix parse error: open ./api: no such file or directory Date: 2021-01-19 12:52:04-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen,build/openrpc: maybe fix no such file error and run 'make docsgen' Date: 2021-01-19 12:55:52-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api/docgen: return if AST comment/groupdoc parsing encounters any error This will returns empty comments/docs maps. This should fix issues like: https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218 where the environment makes file lookups hard or impossible. Date: 2021-01-19 13:04:58-06:00 Signed-off-by: meows <b5c6@protonmail.com> * api: Don't depend on build/ * make: support parallel docsgen * openrpc gen: Use simple build version * methodgen * goimports Co-authored-by: meows <b5c6@protonmail.com>
2021-03-19 18:22:46 +00:00
func (sm *StorageMinerAPI) Discover(ctx context.Context) (apitypes.OpenRPCDocument, error) {
return build.OpenRPCDiscoverJSON_Miner(), nil
}
2021-03-26 05:32:03 +00:00
func (sm *StorageMinerAPI) ComputeProof(ctx context.Context, ssi []builtin.SectorInfo, rand abi.PoStRandomness) ([]builtin.PoStProof, error) {
return sm.Epp.ComputeProof(ctx, ssi, rand)
}
2019-07-24 00:58:31 +00:00
var _ api.StorageMiner = &StorageMinerAPI{}