From b4fada2a4bcc3f0edc0e9a5b277e3ee8b5842d89 Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Fri, 13 Dec 2019 16:05:22 +0100 Subject: [PATCH 1/6] add first pass at documentation on how to use mining worker --- documentation/en/mining-worker.md | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 documentation/en/mining-worker.md diff --git a/documentation/en/mining-worker.md b/documentation/en/mining-worker.md new file mode 100644 index 000000000..d2b2a7268 --- /dev/null +++ b/documentation/en/mining-worker.md @@ -0,0 +1,54 @@ +# Lotus Seal Worker + +The `lotus-seal-worker` is an extra process that can offload heavy processing tasks from your `lotus-storage-miner`. It can be run on the same machine as your `lotus-storage-miner`, or on a different machine communicating over a fast network. + +## Get Started +Make sure that the `lotus-seal-worker` is installed by running: + +```sh +make lotus-seal-worker +``` + +## Running Alongside Storage Miner +You may wish to run the lotus seal worker on the same computer as the storage miner. This allows you to easily set the process priority of the sealing tasks to be lower than the priority of your more important storage miner process. + +To do this, simply run `lotus-seal-worker run`, and the seal worker will automatically pick up the correct authentication tokens from the `LOTUS_STORAGE_PATH` miner repository. + +To check that the seal worker is properly connected to your storage miner, run `lotus-storage-miner info` and check that the remote worker count has increased. + +TODO: sample output + +## Running Over the Network +To use an entirely separate computer for sealing tasks, you will want to run the `lotus-seal-worker` on a separate machine, connected to your storage miner via the local area network. + +This setup is a little more complex than running it locally. + +First, you will need to ensure your `lotus-storage-miner`'s API is accessible over the network. + +To do this, open up `~/.lotusstorage/config.toml` (Or if you manually set `LOTUS_STORAGE_PATH`, look under that directory) and look for the API field. + +By default it should look something like: +```toml +[API] +ListenAddress = "/ip4/127.0.0.1/tcp/2345/http" +``` + +To make your node accessible over the local area network, you will need to determine your machines IP on the LAN, and change the `127.0.0.1` in the file to that address. A less secure, but more permissive option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the API (though they will still need an auth token, as we will discuss next). + +Next, you will need to get an authentication token for the seal worker. All lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them. To create a token, run `lotus-storage-miner auth create-token --perm admin`. This will create a token with `admin` permissions. (TODO: does the seal worker need admin? or can we get away with less?) (if it does need admin powers, insert a warning here about how powerful this token is) + +This token will look something like this: +```sh +why@WhyNet ~> lotus-storage-miner auth create-token --perm admin +eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.KWWdh1jOVP_5YMAp8x5wNomFGgKS75ucOtj1ah5iP7k +``` + +Now that you have allowed the storage miner to be connected to, and have created an auth token, its time to connect up the seal worker. + +On the machine that you will be running the `lotus-seal-worker` on, you will need to set the `STORAGE_API_INFO` environment variable to `TOKEN:STORAGE_NODE_MULTIADDR`. Where `TOKEN` is the token we created above, and `STORAGE_NODE_MULTIADDR` is the multiaddr of the storage miners api that we set in the config file. + +Once this is set, you should be able to just run `lotus-seal-worker run`. + +To check that the seal worker is properly connected to your storage miner, run `lotus-storage-miner info` and check that the remote worker count has increased. + +TODO: sample output From 41ed61c6fec3f71f8cf86c0da0ed44aff5a91d9c Mon Sep 17 00:00:00 2001 From: Whyrusleeping Date: Mon, 16 Dec 2019 17:50:24 +0100 Subject: [PATCH 2/6] Update documentation/en/mining-worker.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Łukasz Magiera --- documentation/en/mining-worker.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/documentation/en/mining-worker.md b/documentation/en/mining-worker.md index d2b2a7268..5c7f69e19 100644 --- a/documentation/en/mining-worker.md +++ b/documentation/en/mining-worker.md @@ -16,7 +16,16 @@ To do this, simply run `lotus-seal-worker run`, and the seal worker will automat To check that the seal worker is properly connected to your storage miner, run `lotus-storage-miner info` and check that the remote worker count has increased. -TODO: sample output +``` +Miner: t0103 +Sector Size: 16.0 MiB +Power: 0 B / 16.0 MiB (0%) +Worker use: + Local: 0 / 2 (+1 reserved) + **Remote: 0 / 1** +PoSt Submissions: Not Proving +Sectors: map[Committing:0 Proving:0 Total:0] +``` ## Running Over the Network To use an entirely separate computer for sealing tasks, you will want to run the `lotus-seal-worker` on a separate machine, connected to your storage miner via the local area network. From 9bdbc5daf1ad82ee986dc89ac422a60f60d1d968 Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Tue, 17 Dec 2019 16:17:45 -0800 Subject: [PATCH 3/6] fix auth command to correctly target storage miner --- cmd/lotus-storage-miner/auth.go | 67 +++++++++++++++++++++++++++++++++ cmd/lotus-storage-miner/main.go | 4 +- 2 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 cmd/lotus-storage-miner/auth.go diff --git a/cmd/lotus-storage-miner/auth.go b/cmd/lotus-storage-miner/auth.go new file mode 100644 index 000000000..757833970 --- /dev/null +++ b/cmd/lotus-storage-miner/auth.go @@ -0,0 +1,67 @@ +package main + +import ( + "errors" + "fmt" + + "gopkg.in/urfave/cli.v2" + + "github.com/filecoin-project/lotus/api/apistruct" + lcli "github.com/filecoin-project/lotus/cli" +) + +var authCmd = &cli.Command{ + Name: "auth", + Usage: "Manage RPC permissions", + Subcommands: []*cli.Command{ + authCreateAdminToken, + }, +} + +var authCreateAdminToken = &cli.Command{ + Name: "create-token", + Usage: "Create token", + Flags: []cli.Flag{ + &cli.StringFlag{ + Name: "perm", + Usage: "permission to assign to the token, one of: read, write, sign, admin", + }, + }, + + Action: func(cctx *cli.Context) error { + napi, closer, err := lcli.GetStorageMinerAPI(cctx) + if err != nil { + return err + } + defer closer() + + ctx := lcli.ReqContext(cctx) + + if !cctx.IsSet("perm") { + return errors.New("--perm flag not set") + } + + perm := cctx.String("perm") + idx := 0 + for i, p := range apistruct.AllPermissions { + if perm == p { + idx = i + 1 + } + } + + if idx == 0 { + return fmt.Errorf("--perm flag has to be one of: %s", apistruct.AllPermissions) + } + + // slice on [:idx] so for example: 'sign' gives you [read, write, sign] + token, err := napi.AuthNew(ctx, apistruct.AllPermissions[:idx]) + if err != nil { + return err + } + + // TODO: Log in audit log when it is implemented + + fmt.Println(string(token)) + return nil + }, +} diff --git a/cmd/lotus-storage-miner/main.go b/cmd/lotus-storage-miner/main.go index 00c170703..61eaa796b 100644 --- a/cmd/lotus-storage-miner/main.go +++ b/cmd/lotus-storage-miner/main.go @@ -8,7 +8,6 @@ import ( "gopkg.in/urfave/cli.v2" "github.com/filecoin-project/lotus/build" - lcli "github.com/filecoin-project/lotus/cli" "github.com/filecoin-project/lotus/node/repo" "github.com/filecoin-project/lotus/tracing" ) @@ -22,6 +21,7 @@ func main() { logging.SetLogLevel("swarm", "WARN") local := []*cli.Command{ + authCmd, runCmd, initCmd, infoCmd, @@ -67,7 +67,7 @@ func main() { }, }, - Commands: append(local, lcli.Commands...), + Commands: local, } app.Setup() app.Metadata["repoType"] = repo.StorageMiner From 2322c42fd0d19e665f5e729fc51dc0cd344208e5 Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Tue, 17 Dec 2019 16:52:17 -0800 Subject: [PATCH 4/6] do the auth command the right way --- cli/auth.go | 2 +- cmd/lotus-storage-miner/auth.go | 67 --------------------------------- cmd/lotus-storage-miner/main.go | 4 +- 3 files changed, 3 insertions(+), 70 deletions(-) delete mode 100644 cmd/lotus-storage-miner/auth.go diff --git a/cli/auth.go b/cli/auth.go index d957881b6..b912dda4b 100644 --- a/cli/auth.go +++ b/cli/auth.go @@ -28,7 +28,7 @@ var authCreateAdminToken = &cli.Command{ }, Action: func(cctx *cli.Context) error { - napi, closer, err := GetFullNodeAPI(cctx) + napi, closer, err := GetAPI(cctx) if err != nil { return err } diff --git a/cmd/lotus-storage-miner/auth.go b/cmd/lotus-storage-miner/auth.go deleted file mode 100644 index 757833970..000000000 --- a/cmd/lotus-storage-miner/auth.go +++ /dev/null @@ -1,67 +0,0 @@ -package main - -import ( - "errors" - "fmt" - - "gopkg.in/urfave/cli.v2" - - "github.com/filecoin-project/lotus/api/apistruct" - lcli "github.com/filecoin-project/lotus/cli" -) - -var authCmd = &cli.Command{ - Name: "auth", - Usage: "Manage RPC permissions", - Subcommands: []*cli.Command{ - authCreateAdminToken, - }, -} - -var authCreateAdminToken = &cli.Command{ - Name: "create-token", - Usage: "Create token", - Flags: []cli.Flag{ - &cli.StringFlag{ - Name: "perm", - Usage: "permission to assign to the token, one of: read, write, sign, admin", - }, - }, - - Action: func(cctx *cli.Context) error { - napi, closer, err := lcli.GetStorageMinerAPI(cctx) - if err != nil { - return err - } - defer closer() - - ctx := lcli.ReqContext(cctx) - - if !cctx.IsSet("perm") { - return errors.New("--perm flag not set") - } - - perm := cctx.String("perm") - idx := 0 - for i, p := range apistruct.AllPermissions { - if perm == p { - idx = i + 1 - } - } - - if idx == 0 { - return fmt.Errorf("--perm flag has to be one of: %s", apistruct.AllPermissions) - } - - // slice on [:idx] so for example: 'sign' gives you [read, write, sign] - token, err := napi.AuthNew(ctx, apistruct.AllPermissions[:idx]) - if err != nil { - return err - } - - // TODO: Log in audit log when it is implemented - - fmt.Println(string(token)) - return nil - }, -} diff --git a/cmd/lotus-storage-miner/main.go b/cmd/lotus-storage-miner/main.go index 61eaa796b..00c170703 100644 --- a/cmd/lotus-storage-miner/main.go +++ b/cmd/lotus-storage-miner/main.go @@ -8,6 +8,7 @@ import ( "gopkg.in/urfave/cli.v2" "github.com/filecoin-project/lotus/build" + lcli "github.com/filecoin-project/lotus/cli" "github.com/filecoin-project/lotus/node/repo" "github.com/filecoin-project/lotus/tracing" ) @@ -21,7 +22,6 @@ func main() { logging.SetLogLevel("swarm", "WARN") local := []*cli.Command{ - authCmd, runCmd, initCmd, infoCmd, @@ -67,7 +67,7 @@ func main() { }, }, - Commands: local, + Commands: append(local, lcli.Commands...), } app.Setup() app.Metadata["repoType"] = repo.StorageMiner From 5732e4f2b784b26bccc9eef8d40f4983eea16159 Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Tue, 17 Dec 2019 17:19:46 -0800 Subject: [PATCH 5/6] address review feedback on doc --- documentation/en/mining-worker.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/documentation/en/mining-worker.md b/documentation/en/mining-worker.md index 5c7f69e19..9c3ef5e8d 100644 --- a/documentation/en/mining-worker.md +++ b/documentation/en/mining-worker.md @@ -17,6 +17,7 @@ To do this, simply run `lotus-seal-worker run`, and the seal worker will automat To check that the seal worker is properly connected to your storage miner, run `lotus-storage-miner info` and check that the remote worker count has increased. ``` +why@computer ~/lotus> lotus-storage-miner info Miner: t0103 Sector Size: 16.0 MiB Power: 0 B / 16.0 MiB (0%) @@ -44,11 +45,11 @@ ListenAddress = "/ip4/127.0.0.1/tcp/2345/http" To make your node accessible over the local area network, you will need to determine your machines IP on the LAN, and change the `127.0.0.1` in the file to that address. A less secure, but more permissive option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the API (though they will still need an auth token, as we will discuss next). -Next, you will need to get an authentication token for the seal worker. All lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them. To create a token, run `lotus-storage-miner auth create-token --perm admin`. This will create a token with `admin` permissions. (TODO: does the seal worker need admin? or can we get away with less?) (if it does need admin powers, insert a warning here about how powerful this token is) +Next, you will need to get an authentication token for the seal worker. All lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them. To create a token, run `lotus-storage-miner auth create-token --perm admin`. This will create a token with `admin` permissions. Note: This is an admin token, it can access any of the api endpoints of your node, take care not to leak it. This token will look something like this: ```sh -why@WhyNet ~> lotus-storage-miner auth create-token --perm admin +why@computer ~> lotus-storage-miner auth create-token --perm admin eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.KWWdh1jOVP_5YMAp8x5wNomFGgKS75ucOtj1ah5iP7k ``` @@ -60,4 +61,15 @@ Once this is set, you should be able to just run `lotus-seal-worker run`. To check that the seal worker is properly connected to your storage miner, run `lotus-storage-miner info` and check that the remote worker count has increased. -TODO: sample output +``` +why@computer ~/lotus> lotus-storage-miner info +Miner: t05749 +Sector Size: 1 GiB +Power: 0 B / 136 TiB (0.0000%) + Committed: 1 GiB + Proving: 1 GiB +Worker use: + Local: 0 / 1 (+1 reserved) + **Remote: 0 / 1** +Sectors: map[Proving:1 Total:1] +``` From 1243ddfdf2f6e344f6a482646dd7ecdd22d1ffed Mon Sep 17 00:00:00 2001 From: whyrusleeping Date: Tue, 17 Dec 2019 17:21:20 -0800 Subject: [PATCH 6/6] link to api auth doc --- documentation/en/mining-worker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/en/mining-worker.md b/documentation/en/mining-worker.md index 9c3ef5e8d..a92e2ab5d 100644 --- a/documentation/en/mining-worker.md +++ b/documentation/en/mining-worker.md @@ -45,7 +45,7 @@ ListenAddress = "/ip4/127.0.0.1/tcp/2345/http" To make your node accessible over the local area network, you will need to determine your machines IP on the LAN, and change the `127.0.0.1` in the file to that address. A less secure, but more permissive option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the API (though they will still need an auth token, as we will discuss next). -Next, you will need to get an authentication token for the seal worker. All lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them. To create a token, run `lotus-storage-miner auth create-token --perm admin`. This will create a token with `admin` permissions. Note: This is an admin token, it can access any of the api endpoints of your node, take care not to leak it. +Next, you will need to get an authentication token for the seal worker. All lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them. To create a token, run `lotus-storage-miner auth create-token --perm admin`. This will create a token with `admin` permissions. Note: This is an admin token, it can access any of the api endpoints of your node, take care not to leak it. See the [scripting support doc](api-scripting-support.md) for more details. This token will look something like this: ```sh