documentation: removes old docs folder, adds new documentation folder with organized content

docs/API.md

@@ -1,32 +0,0 @@
-TODO: make this into a nicer doc
-
-### Endpoints
-
-By default `127.0.0.1:1234` - daemon stores the api endpoint multiaddr in `~/.lotus/api`
-
-* `http://[api:port]/rpc/v0` - JsonRPC http endpoint
-* `ws://[api:port]/rpc/v0` - JsonRPC websocket endpoint
-* `PUT http://[api:port]/rest/v0/import` - import file to the node repo
-  * Requires write permission
-
-For JsonRPC interface definition see `api/api.go`. Required permissions are
-defined in `api/struct.go`
-
-### Auth:
-
-JWT in the `Authorization: Bearer <token>` http header
-
-Permissions:
-* `read` - Read node state, no private data
-* `write` - Write to local store / chain, read private data
-* `sign` - Use private keys stored in wallet for signing
-* `admin` - Manage permissions
-
-Payload:
-```json
-{
-  "Allow": ["read", "write", ...]
-}
-```
-
-Admin token is stored in `~/.lotus/token`

docs/images/lotus_logo_h.svg

@@ -1 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 206.62 84.875"><defs><style>.a{fill:#716ee0;}.b{fill:#2cdbbe;}.c{fill:#4df4e0;}.d{fill:#807cf7;}.e,.n{fill:#5c57bc;}.f{fill:#18b4c4;}.g{fill:#27ba9e;}.h{fill:#169fa5;}.i{fill:#27baba;}.j{fill:#5d5dce;}.k{fill:#4545a3;}.l{fill:#6969e2;}.m{fill:#fff;}.n{stroke:#5c57bc;stroke-miterlimit:10;stroke-width:0.5px;}</style></defs><title>lotus_logo_h</title><path class="a" d="M76.1,66.251c-13.29,0-24.065-14.125-24.065-14.125S62.807,38,76.1,38s24.05,14.125,24.05,14.125S89.387,66.251,76.1,66.251Z" transform="translate(-2.582 -9.691)"/><path class="b" d="M26.647,66.251c-13.29,0-24.065-14.125-24.065-14.125S13.347,38,26.647,38,50.712,52.126,50.712,52.126,39.927,66.251,26.647,66.251Z" transform="translate(-2.582 -9.691)"/><path class="c" d="M26.647,38.011c6.95,11.56,24.385,13.54,24.385,13.54s6.155-16.76-.795-28.32S25.852,9.691,25.852,9.691,19.682,26.451,26.647,38.011Z" transform="translate(-2.582 -9.691)"/><path class="d" d="M52.507,81.026c6.955,11.56,24.385,13.54,24.385,13.54s6.16-16.76-.8-28.32-24.385-13.54-24.385-13.54S45.557,69.466,52.507,81.026Z" transform="translate(-2.582 -9.691)"/><path class="e" d="M76.1,38.011c-6.95,11.56-24.385,13.54-24.385,13.54s-6.155-16.76.795-28.32S76.892,9.691,76.892,9.691,83.052,26.451,76.1,38.011Z" transform="translate(-2.582 -9.691)"/><path class="f" d="M50.227,81.026c-6.955,11.56-24.385,13.54-24.385,13.54s-6.16-16.76.8-28.32,24.385-13.54,24.385-13.54S57.177,69.466,50.227,81.026Z" transform="translate(-2.582 -9.691)"/><path class="g" d="M51.607,50.421c1.075-3.565,4.115-15.55-.25-25.075C47.182,34.976,49.862,46.766,51.607,50.421Z" transform="translate(-2.582 -9.691)"/><path class="h" d="M26.647,66.246h0c11.5-.12,21-10.79,23.355-13.66C46.387,53.206,32.557,56.181,26.647,66.246Z" transform="translate(-2.582 -9.691)"/><path class="i" d="M26.647,38.006h0c6.115,9.94,19.805,13.055,23.63,13.17C47.867,48.351,38.152,37.881,26.652,38Z" transform="translate(-2.582 -9.691)"/><path class="j" d="M52.487,53.076C54.9,55.9,64.612,66.371,76.112,66.251h0C70,56.306,56.472,53.076,52.487,53.076Z" transform="translate(-2.582 -9.691)"/><path class="k" d="M52.757,51.666c3.615-.62,17.44-3.59,23.36-13.655h0C64.612,38.121,55.107,48.791,52.757,51.666Z" transform="translate(-2.582 -9.691)"/><path class="l" d="M51.372,78.826c4.175-9.61,1.275-21.765-.25-25.075C50.047,57.316,47.007,69.306,51.372,78.826Z" transform="translate(-2.582 -9.691)"/><path class="m" d="M51.3,35.586a16.545,16.545,0,1,0,16.435,16.68A16.545,16.545,0,0,0,51.3,35.586Z" transform="translate(-2.582 -9.691)"/><path class="n" d="M58.687,43.336a.63.63,0,0,1-.775.04,1.032,1.032,0,0,1-.24-.185,5.767,5.767,0,0,1-.6-.75A1.181,1.181,0,0,0,56,41.921a1.1,1.1,0,0,0-.665.355,4.14,4.14,0,0,0-1,1.5c-.545,1.675-1,3.39-1.5,5.18l4.38.65c-.12.42-.22.785-.34,1.215h0l-4.235-.6-.5,2.5,4.545.665-.35,1.225h0l-.75-.11-2.935-.425-.75-.105c-.28,1.065-.5,2.11-.84,3.125a25.509,25.509,0,0,1-1.28,3.29,4.525,4.525,0,0,1-3.25,2.58,3.936,3.936,0,0,1-.675.08,2.805,2.805,0,0,1-1.88-.57,1.191,1.191,0,0,1-.5-.725,1.162,1.162,0,0,1,.355-.86.311.311,0,0,1,.16-.05,1.091,1.091,0,0,1,.64.17,3.7,3.7,0,0,1,.655.84,1.129,1.129,0,0,0,1.05.54,1.312,1.312,0,0,0,.7-.33,4.506,4.506,0,0,0,1.365-2.4c.5-1.88.915-3.78,1.36-5.675a2.6,2.6,0,0,0,0-.33l-4.21-.645c.05-.41.1-.76.16-1.22h0l4.345.605.59-2.5-4.55-.7c.055-.42.11-.815.165-1.24h0l4.655.675c.125-.46.235-.885.36-1.3A13.054,13.054,0,0,1,53.007,43.2a4.22,4.22,0,0,1,3.7-2h0a2.584,2.584,0,0,1,1.925.81c.105.12.255.27.25.4C58.872,42.731,58.882,43.156,58.687,43.336Z" transform="translate(-2.582 -9.691)"/><path d="M120.3,67.741c-3.73,0-5.5-1.975-5.5-5.405V36h5.5V63.366h2.96v4.375Z" transform="translate(-2.582 -9.691)"/><path d="M125.922,56.5c0-7.16,4-11.665,10.38-11.665s10.425,4.5,10.425,11.665-4,11.755-10.425,11.755S125.922,63.711,125.922,56.5Zm15.055,2.105v-4.16c0-3.35-1.8-5.19-4.675-5.19s-4.655,1.84-4.655,5.19v4.16c0,3.385,1.805,5.23,4.635,5.23s4.695-1.845,4.695-5.23Z" transform="translate(-2.582 -9.691)"/><path d="M152.6,62.036V49.726h-3.3V45.351h1.715c1.63,0,2.145-.775,2.145-2.315v-3.82h4.93v6.135h4.59v4.375h-4.59v13.64h4.25v4.375h-3.945C154.617,67.741,152.6,65.681,152.6,62.036Z" transform="translate(-2.582 -9.691)"/><path d="M180.822,64.011h-.215a6.073,6.073,0,0,1-6.26,4.245c-4.59,0-7.165-3.175-7.165-8.75V45.351h5.5v13.6c0,3.09,1.16,4.765,3.775,4.765,2.185,0,4.375-1.16,4.375-3.56v-14.8h5.5v22.39h-5.5Z" transform="translate(-2.582 -9.691)"/><path d="M190.727,64.136l3.345-3.26a7.863,7.863,0,0,0,6.135,3.045c2.615,0,3.73-1,3.73-2.57,0-1.29-.77-2.1-2.745-2.36l-2.23-.3c-4.8-.6-7.46-2.705-7.46-6.78,0-4.33,3.385-7.075,8.5-7.075,4.375,0,6.6,1.285,8.75,3.5l-3.215,3.22a7.4,7.4,0,0,0-5.39-2.39c-2.355,0-3.385.945-3.385,2.32,0,1.5.815,2.185,2.915,2.53l2.275.3c4.89.685,7.25,2.875,7.25,6.6,0,4.33-3.65,7.335-9.225,7.335A11.357,11.357,0,0,1,190.727,64.136Z" transform="translate(-2.582 -9.691)"/></svg>

documentation/api.md

@@ -0,0 +1,35 @@
+# API
+
+The systems API is defined in here. The RPC maps directly to the API defined here using the JSON RPC package in `lib/jsonrpc`.
+
+## Overview
+
+By default `127.0.0.1:1234` - daemon stores the api endpoint multiaddr in `~/.lotus/api`
+
+- `http://[api:port]/rpc/v0` - JsonRPC http endpoint
+- `ws://[api:port]/rpc/v0` - JsonRPC websocket endpoint
+- `PUT http://[api:port]/rest/v0/import` - import file to the node repo, it requires write permission.
+
+For JsonRPC interface definition see `api/api.go`. Required permissions are
+defined in `api/struct.go`
+
+## Auth
+
+JWT in the `Authorization: Bearer <token>` http header
+
+Permissions
+
+- `read` - Read node state, no private data
+- `write` - Write to local store / chain, read private data
+- `sign` - Use private keys stored in wallet for signing
+- `admin` - Manage permissions
+
+Payload
+
+```json
+{
+  "Allow": ["read", "write", ...]
+}
+```
+
+Admin token is stored in `~/.lotus/token`

documentation/getting-started.md

@@ -0,0 +1,9 @@
+# Project Lotus - 莲
+
+Lotus is an experimental implementation of the **Filecoin Distributed Storage Network**. For more details about Filecoin, check out the [Filecoin Spec](https://github.com/filecoin-project/specs).
+
+## Background
+
+Lotus is architected modularly, and aims to keep clean API boundaries between everything, even if they are in the same process. Notably, the 'lotus full node' software, and the 'lotus storage miner' software are two separate programs.
+
+The **Lotus Storage Miner** is intended to be run on the machine that manages a single storage miner instance, and is meant to communicate with the full node via the websockets JSON RPC API for all of its chain interaction needs. This way, a mining operation may easily run one or many storage miners, connected to one or many full node instances.

documentation/glossary.md

@@ -0,0 +1,33 @@
+# Glossary
+
+## Chain/Types
+
+Implementation of data structures used by Filecoin and their serializations.
+
+## Chain/Store
+
+The chainstore manages all local chain state, including block headers, messages, and state.
+
+## Chain/State
+
+A package for dealing with the Filecoin state tree. Wraps the [HAMT](https://github.com/ipfs/go-hamt-ipld).
+
+## Chain/Actors
+
+Implementations of the builtin Filecoin network actors.
+
+## Chain/Vm
+
+The Filecoin state machine 'vm'. Implemented here are utilities to invoke Filecoin actor methods.
+
+## Miner
+
+The block producer logic. This package interfaces with the full node through the API, despite currently being implemented in the same process (very likely to be extracted as its own separate process in the near future).
+
+## Storage
+
+The storage miner logic. This package also interfaces with the full node through a subset of the api. This code is used to implement the `lotus-storage-miner` process.
+
+## Pond
+
+Pond is a graphical testbed for lotus. It can be used to spin up nodes, connect them in a given topology, start them mining, and observe how they function over time.

documentation/hardware.md

@@ -0,0 +1,3 @@
+# Picking Hardware
+
+Coming soon

documentation/install-lotus-arch.md

@@ -0,0 +1,44 @@
+# Installing Lotus on Arch Linux
+
+Install these dependencies:
+
+- go (1.13 or higher)
+- gcc (7.4.0 or higher)
+- git (version 2 or higher)
+- bzr (some go dependency needs this)
+- jq
+- pkg-config
+- opencl-icd-loader
+- opencl driver (like nvidia-opencl on arch) (for GPU acceleration)
+- opencl-headers (build)
+- rustup (proofs build)
+- llvm (proofs build)
+- clang (proofs build)
+
+Arch (run):
+
+```sh
+sudo pacman -Syu opencl-icd-loader
+```
+
+Arch (build):
+
+```sh
+sudo pacman -Syu go gcc git bzr jq pkg-config opencl-icd-loader opencl-headers
+```
+
+## Clone
+
+```sh
+$ git clone https://github.com/filecoin-project/lotus.git
+$ cd lotus/
+```
+
+## Install
+
+```sh
+$ make clean all
+$ sudo make install
+```
+
+Now you can use the command `lotus` in the command line.

documentation/install-lotus-macos.md

@@ -0,0 +1,32 @@
+# Installing Lotus on MacOS
+
+Install these dependencies:
+
+- go (1.13 or higher)
+- gcc (7.4.0 or higher)
+- git (version 2 or higher)
+- bzr (some go dependency needs this)
+- jq
+- pkg-config
+- opencl-icd-loader
+- opencl driver (like nvidia-opencl on arch) (for GPU acceleration)
+- opencl-headers (build)
+- rustup (proofs build)
+- llvm (proofs build)
+- clang (proofs build)
+
+## Clone
+
+```sh
+$ git clone https://github.com/filecoin-project/lotus.git
+$ cd lotus/
+```
+
+## Make & Install
+
+```sh
+$ make clean all
+$ sudo make install
+```
+
+Now you can use the command `lotus` in the command line.

documentation/install-lotus-ubuntu.md

@@ -0,0 +1,47 @@
+# Installing Lotus on Ubuntu
+
+Install these dependencies:
+
+- go (1.13 or higher)
+- gcc (7.4.0 or higher)
+- git (version 2 or higher)
+- bzr (some go dependency needs this)
+- jq
+- pkg-config
+- opencl-icd-loader
+- opencl driver (like nvidia-opencl on arch) (for GPU acceleration)
+- opencl-headers (build)
+- rustup (proofs build)
+- llvm (proofs build)
+- clang (proofs build)
+
+Ubuntu / Debian (run):
+
+```sh
+sudo apt update
+sudo apt install mesa-opencl-icd ocl-icd-opencl-dev
+```
+
+Ubuntu (build):
+
+```sh
+sudo add-apt-repository ppa:longsleep/golang-backports
+sudo apt update
+sudo apt install golang-go gcc git bzr jq pkg-config mesa-opencl-icd ocl-icd-opencl-dev
+```
+
+## Clone
+
+```sh
+$ git clone https://github.com/filecoin-project/lotus.git
+$ cd lotus/
+```
+
+## Install
+
+```sh
+$ make clean all
+$ sudo make install
+```
+
+Now you can use the command `lotus` in the command line.

documentation/mining.md

@@ -0,0 +1,53 @@
+# Mining
+
+Ensure that at least one BLS address (`t3..`) in your wallet exists
+
+```sh
+$ lotus wallet list
+t3...
+```
+
+With this address, go to https://lotus-faucet.kittyhawk.wtf/miner.html, and
+click `Create Miner`
+
+Wait for a page telling you the address of the newly created storage miner to
+appear - It should be saying: `New storage miners address is: t0..`
+
+Initialize storage miner:
+
+```sh
+$ lotus-storage-miner init --actor=t01.. --owner=t3....
+```
+
+This command should return successfully after miner is setup on-chain (30-60s)
+
+Start mining:
+
+```sh
+$ lotus-storage-miner run
+```
+
+To view the miner id used for deals:
+
+```sh
+$ lotus-storage-miner info
+```
+
+e.g. miner id `t0111`
+
+Seal random data to start producing PoSts:
+
+```sh
+$ lotus-storage-miner store-garbage
+```
+
+You can check miner power and sector usage with the miner id:
+
+```sh
+# Total power of the network
+$ lotus-storage-miner state power
+
+$ lotus-storage-miner state power <miner>
+
+$ lotus-storage-miner state sectors <miner>
+```

documentation/retrieving-data.md

@@ -0,0 +1,17 @@
+# Retrieving Data
+
+If you have stored data with a miner in the network, you can search for it by CID
+
+```sh
+$ lotus client find <Data CID>
+LOCAL
+RETRIEVAL <miner>@<miner peerId>-<deal funds>-<size>
+```
+
+Retrieve data from a miner
+
+```sh
+\$ lotus client retrieve <Data CID> <outfile>
+```
+
+This will initiate a retrieval deal and write the data to the outfile. This process may take some time.

documentation/setup-devnet-cli.md

@@ -0,0 +1,75 @@
+# Setup Lotus Devnet using the CLI
+
+## Node setup
+
+If you have run lotus before and want to remove all previous data: `rm -rf ~/.lotus ~/.lotusstorage`
+
+## Genesis & Bootstrap
+
+The current lotus build will automatically join the lotus Devnet using the genesis and bootstrap files in the `build/` directory. No configuration is needed.
+
+## Start Daemon
+
+```sh
+$ lotus daemon
+```
+
+In another window check that you are connected to the network:
+
+```sh
+$ lotus net peers | wc -l
+2 # number of peers
+```
+
+Wait for the chain to finish syncing:
+
+```sh
+$ lotus sync wait
+```
+
+You can view latest block height along with other network metrics at the https://lotus-metrics.kittyhawk.wtf/chain.
+
+## Basics
+
+Create a new address:
+
+```sh
+$ lotus wallet new bls
+t3...
+```
+
+Grab some funds from faucet - go to https://lotus-faucet.kittyhawk.wtf/, paste the address
+you just created, and press Send.
+
+Check the wallet balance (balance is listed in attoFIL, where 1 attoFIL = 10^-18 FIL):
+
+```sh
+$ lotus wallet balance [optional address (t3...)]
+```
+
+If you see an error like `actor not found` after executing this command, it means that either your node isn't fully synced or there are no transactions to this address yet on chain. If the latter, using the faucet should 'fix' this.
+
+## Make a deal
+
+It is possible for a Client to make a deal with a Miner on the same lotus Node.
+
+```sh
+
+# List all miners in the system. Choose one to make a deal with.
+
+$ lotus state list-miners
+
+# List asks proposed by a miner
+
+$ lotus client query-ask <miner>
+
+# Propose a deal with a miner. Price is in attoFIL/byte/block. Duration is # of blocks.
+
+$ lotus client deal <Data CID> <miner> <price> <duration>
+```
+
+For example `\$ lotus client deal bafkre...qvtjsi t0111 36000 12` proposes a deal to store CID `bafkre...qvtjsi` with miner `t0111` at price `36000` for a duration of `12` blocks. If successful, the `client deal` command will return a deal CID.
+
+## Monitoring Dashboard
+
+To see the latest network activity, including chain block height, blocktime, total network power, largest miners, and more, check out the monitoring dashboard at https://lotus-metrics.kittyhawk.wtf.

documentation/setup-devnet-gui.md

@@ -0,0 +1,34 @@
+# Setup Lotus Devnet using a GUI
+
+As an alternative to the CLI you can use Pond, a graphical testbed for lotus. It can be used to spin up nodes, connect them in a given topology, start them mining, and observe how they function over time.
+
+Build:
+
+```sh
+$ make pond
+```
+
+Run:
+
+```sh
+$ ./pond run
+Listening on http://127.0.0.1:2222
+```
+
+Now go to http://127.0.0.1:2222.
+
+## Thing to try
+
+- The `Spawn Node` button starts a new lotus Node in a new draggable window.
+- Click `[Spawn Storage Miner]` to start mining (make sure the Node's wallet has funds).
+- Click on `[Client]` to open the Node's client interface and propose a deal with an existing Miner. If successful you'll see a payment channel open up with that Miner.
+
+Don't leave Pond unattended for long periods of time (10h+), the web-ui tends to eventually consume all the available RAM.
+
+## Troubleshooting
+
+- Turn it off and on - Start at the top
+- `rm -rf ~/.lotus ~/.lotusstorage/`
+- Verify you have the correct versions of dependencies
+- If stuck on a bad fork, try `lotus chain sethead --genesis`
+- If that didn't help, open a new issue, ask in the [Community forum](https://discuss.filecoin.io) or reach out via [Community chat](https://github.com/filecoin-project/community#chat).

documentation/storing-data.md

@@ -0,0 +1,20 @@
+# Storing Data
+
+Start by creating a file, in this example we will use the command line to create `hello.txt`.
+
+```sh
+$ echo "Hi my name is $USER" > hello.txt
+```
+
+Afterwards you can import the file into lotus & get a **Data CID** as output.
+
+```sh
+$ lotus client import ./hello.txt
+<Data CID>
+```
+
+To see a list of files by `CID`, `name`, `size`, `status`.
+
+```sh
+$ lotus client local
+```

documentation/tracing.md

@@ -1,4 +1,8 @@
-## Tracing
+# Tracing
+
+Lotus has tracing built into many of its internals. To view the traces, first download [jaeger](https://www.jaegertracing.io/download/) (Choose the 'all-in-one' binary). Then run it somewhere, start up the lotus daemon, and open up localhost:16686 in your browser.
+
+## Open Census
 
 Lotus uses [OpenCensus](https://opencensus.io/) for tracing application flow.
 This generates spans
@@ -22,6 +26,7 @@ Now, to view any generated traces, open up `http://localhost:16686/` in your
 browser.
 
 ## Adding Spans
+
 To annotate a new codepath with spans, add the following lines to the top of the function you wish to trace:
 
 ```go