cerc-io/lotus
16
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 27 Wiki Activity

Merge branch 'master' into post-trace

Browse Source
This commit is contained in:
Łukasz Magiera 2020-09-30 10:03:54 +02:00 committed by GitHub
commit 90be8d801b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
106 changed files with 4238 additions and 2857 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

57
CHANGELOG.md
View File

@ -1,5 +1,62 @@
# Lotus changelog
# 0.8.0 / 2020-09-26
This consensus-breaking release of Lotus introduces an upgrade to the network. The changes that break consensus are:
- Upgrading to specs-actors v0.9.11, which reduces WindowPoSt faults per [FIP 0002](https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0002.md) to reduce cost for honest miners with occasional faults (see https://github.com/filecoin-project/specs-actors/pull/1181)
- Revisions to some cryptoeconomics and network params
This release also updates go-fil-markets to fix an incompatibility issue between v0.7.2 and earlier versions.
## Changes
#### Dependencies
- Update spec actors to 0.9.11 (https://github.com/filecoin-project/lotus/pull/4039)
- Update markets to 0.6.3 (https://github.com/filecoin-project/lotus/pull/4013)
#### Core Lotus
- Network upgrade (https://github.com/filecoin-project/lotus/pull/4039)
- Fix AddSupportedProofTypes (https://github.com/filecoin-project/lotus/pull/4033)
- Return an error when we fail to find a sector when checking sector expiration (https://github.com/filecoin-project/lotus/pull/4026)
- Batch blockstore copies after block validation (https://github.com/filecoin-project/lotus/pull/3980)
- Remove a misleading miner actor abstraction (https://github.com/filecoin-project/lotus/pull/3977)
- Fix out-of-bounds when loading all sector infos (https://github.com/filecoin-project/lotus/pull/3976)
- Fix break condition in the miner (https://github.com/filecoin-project/lotus/pull/3953)
#### UX
- Correct helptext around miners setting ask (https://github.com/filecoin-project/lotus/pull/4009)
- Make sync wait nicer (https://github.com/filecoin-project/lotus/pull/3991)
#### Tooling and validation
- Small adjustments following network upgradability changes (https://github.com/filecoin-project/lotus/pull/3996)
- Add some more big pictures stats to stateroot stat (https://github.com/filecoin-project/lotus/pull/3995)
- Add some actors policy setters for testing (https://github.com/filecoin-project/lotus/pull/3975)
## Contributors
The following contributors had 5 or more commits go into this release.
We are grateful for every contribution!
| Contributor | Commits | Lines ± |
|--------------------|---------|---------------|
| arajasek | 66 | +3140/-1261 |
| Stebalien | 64 | +3797/-3434 |
| magik6k | 48 | +1892/-976 |
| raulk | 40 | +2412/-1549 |
| vyzo | 22 | +287/-196 |
| alanshaw | 15 | +761/-146 |
| whyrusleeping | 15 | +736/-52 |
| hannahhoward | 14 | +1237/837- |
| anton | 6 | +32/-8 |
| travisperson | 5 | +502/-6 |
| Frank | 5 | +78/-39 |
| Jennifer | 5 | +148/-41 |
# 0.7.2 / 2020-09-23
This optional release of Lotus introduces a major refactor around how a Lotus node interacts with code from the specs-actors repo. We now use interfaces to read the state of actors, which is required to be able to reason about different versions of actors code at the same time.

4
README.md
View File

@ -1,5 +1,5 @@
<p align="center">
<a href="https://lotu.sh/" title="Lotus Docs">
<a href="https://docs.filecoin.io/" title="Filecoin Docs">
<img src="documentation/images/lotus_logo_h.png" alt="Project Lotus Logo" width="244" />
</a>
</p>
@ -18,7 +18,7 @@ Lotus is an implementation of the Filecoin Distributed Storage Network. For more
## Building & Documentation
For instructions on how to build lotus from source, please visit [Lotus build and setup instruction](https://docs.filecoin.io/get-started/lotus/installation/#minimal-requirements) or read the source [here](https://github.com/filecoin-project/lotus/tree/master/documentation).
For instructions on how to build, install and setup lotus, please visit [https://docs.filecoin.io/get-started/lotus](https://docs.filecoin.io/get-started/lotus/).
## Reporting a Vulnerability

5
api/api_full.go
View File

@ -176,6 +176,9 @@ type FullNode interface {
// the reason.
SyncCheckBad(ctx context.Context, bcid cid.Cid) (string, error)
// SyncValidateTipset indicates whether the provided tipset is valid or not
SyncValidateTipset(ctx context.Context, tsk types.TipSetKey) (bool, error)
// MethodGroup: Mpool
// The Mpool methods are for interacting with the message pool. The message pool
// manages all incoming and outgoing 'messages' going over the network.
@ -244,6 +247,8 @@ type FullNode interface {
WalletImport(context.Context, *types.KeyInfo) (address.Address, error)
// WalletDelete deletes an address from the wallet.
WalletDelete(context.Context, address.Address) error
// WalletValidateAddress validates whether a given string can be decoded as a well-formed address
WalletValidateAddress(context.Context, string) (address.Address, error)
// Other

34
api/apistruct/struct.go
View File

@ -112,6 +112,7 @@ type FullNodeStruct struct {
SyncMarkBad func(ctx context.Context, bcid cid.Cid) error `perm:"admin"`
SyncUnmarkBad func(ctx context.Context, bcid cid.Cid) error `perm:"admin"`
SyncCheckBad func(ctx context.Context, bcid cid.Cid) (string, error) `perm:"read"`
SyncValidateTipset func(ctx context.Context, tsk types.TipSetKey) (bool, error) `perm:"read"`
MpoolGetConfig func(context.Context) (*types.MpoolConfig, error) `perm:"read"`
MpoolSetConfig func(context.Context, *types.MpoolConfig) error `perm:"write"`
@ -129,18 +130,19 @@ type FullNodeStruct struct {
MinerGetBaseInfo func(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*api.MiningBaseInfo, error) `perm:"read"`
MinerCreateBlock func(context.Context, *api.BlockTemplate) (*types.BlockMsg, error) `perm:"write"`
WalletNew func(context.Context, crypto.SigType) (address.Address, error) `perm:"write"`
WalletHas func(context.Context, address.Address) (bool, error) `perm:"write"`
WalletList func(context.Context) ([]address.Address, error) `perm:"write"`
WalletBalance func(context.Context, address.Address) (types.BigInt, error) `perm:"read"`
WalletSign func(context.Context, address.Address, []byte) (*crypto.Signature, error) `perm:"sign"`
WalletSignMessage func(context.Context, address.Address, *types.Message) (*types.SignedMessage, error) `perm:"sign"`
WalletVerify func(context.Context, address.Address, []byte, *crypto.Signature) (bool, error) `perm:"read"`
WalletDefaultAddress func(context.Context) (address.Address, error) `perm:"write"`
WalletSetDefault func(context.Context, address.Address) error `perm:"admin"`
WalletExport func(context.Context, address.Address) (*types.KeyInfo, error) `perm:"admin"`
WalletImport func(context.Context, *types.KeyInfo) (address.Address, error) `perm:"admin"`
WalletDelete func(context.Context, address.Address) error `perm:"write"`
WalletNew func(context.Context, crypto.SigType) (address.Address, error) `perm:"write"`
WalletHas func(context.Context, address.Address) (bool, error) `perm:"write"`
WalletList func(context.Context) ([]address.Address, error) `perm:"write"`
WalletBalance func(context.Context, address.Address) (types.BigInt, error) `perm:"read"`
WalletSign func(context.Context, address.Address, []byte) (*crypto.Signature, error) `perm:"sign"`
WalletSignMessage func(context.Context, address.Address, *types.Message) (*types.SignedMessage, error) `perm:"sign"`
WalletVerify func(context.Context, address.Address, []byte, *crypto.Signature) (bool, error) `perm:"read"`
WalletDefaultAddress func(context.Context) (address.Address, error) `perm:"write"`
WalletSetDefault func(context.Context, address.Address) error `perm:"admin"`
WalletExport func(context.Context, address.Address) (*types.KeyInfo, error) `perm:"admin"`
WalletImport func(context.Context, *types.KeyInfo) (address.Address, error) `perm:"admin"`
WalletDelete func(context.Context, address.Address) error `perm:"write"`
WalletValidateAddress func(context.Context, string) (address.Address, error) `perm:"read"`
ClientImport func(ctx context.Context, ref api.FileRef) (*api.ImportRes, error) `perm:"admin"`
ClientListImports func(ctx context.Context) ([]api.Import, error) `perm:"write"`
@ -631,6 +633,10 @@ func (c *FullNodeStruct) WalletDelete(ctx context.Context, addr address.Address)
return c.Internal.WalletDelete(ctx, addr)
}
func (c *FullNodeStruct) WalletValidateAddress(ctx context.Context, str string) (address.Address, error) {
return c.Internal.WalletValidateAddress(ctx, str)
}
func (c *FullNodeStruct) MpoolGetNonce(ctx context.Context, addr address.Address) (uint64, error) {
return c.Internal.MpoolGetNonce(ctx, addr)
}
@ -735,6 +741,10 @@ func (c *FullNodeStruct) SyncCheckBad(ctx context.Context, bcid cid.Cid) (string
return c.Internal.SyncCheckBad(ctx, bcid)
}
func (c *FullNodeStruct) SyncValidateTipset(ctx context.Context, tsk types.TipSetKey) (bool, error) {
return c.Internal.SyncValidateTipset(ctx, tsk)
}
func (c *FullNodeStruct) StateNetworkName(ctx context.Context) (dtypes.NetworkName, error) {
return c.Internal.StateNetworkName(ctx)
}

16
api/test/ccupgrade.go
View File

@ -94,6 +94,22 @@ func TestCCUpgrade(t *testing.T, b APIBuilder, blocktime time.Duration) {
require.Less(t, 50000, int(exp.OnTime))
}
dlInfo, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
require.NoError(t, err)
// Sector should expire.
for {
// Wait for the sector to expire.
status, err := miner.SectorsStatus(ctx, CC, true)
require.NoError(t, err)
if status.OnTime == 0 && status.Early == 0 {
break
}
t.Log("waiting for sector to expire")
// wait one deadline per loop.
time.Sleep(time.Duration(dlInfo.WPoStChallengeWindow) * blocktime)
}
fmt.Println("shutting down mining")
atomic.AddInt64(&mine, -1)
<-done

6
build/bootstrap/bootstrappers.pi
View File

@ -4,3 +4,9 @@
/dns4/bootstrap-4.testnet.fildev.network/tcp/1347/p2p/12D3KooWPkL9LrKRQgHtq7kn9ecNhGU9QaziG8R5tX8v9v7t3h34
/dns4/bootstrap-3.testnet.fildev.network/tcp/1347/p2p/12D3KooWKYSsbpgZ3HAjax5M1BXCwXLa6gVkUARciz7uN3FNtr7T
/dns4/bootstrap-5.testnet.fildev.network/tcp/1347/p2p/12D3KooWQYzqnLASJAabyMpPb1GcWZvNSe7JDcRuhdRqonFoiK9W
/dns4/lotus-bootstrap.forceup.cn/tcp/41778/p2p/12D3KooWFQsv3nRMUevZNWWsY1Wu6NUzUbawnWU5NcRhgKuJA37C
/dns4/bootstrap-0.starpool.in/tcp/12757/p2p/12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz
/dns4/bootstrap-1.starpool.in/tcp/12757/p2p/12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u
/dns4/node.glif.io/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt
/dns4/bootstrap-0.ipfsmain.cn/tcp/34721/p2p/12D3KooWQnwEGNqcM2nAcPtRR9rAX8Hrg4k9kJLCHoTR5chJfz6d
/dns4/bootstrap-1.ipfsmain.cn/tcp/34723/p2p/12D3KooWMKxMkD5DMpSWsW7dBddKxKT7L2GgbNuckz9otxvkvByP

2
build/params_2k.go
View File

@ -12,6 +12,8 @@ const UpgradeBreezeHeight = -1
const BreezeGasTampingDuration = 0
const UpgradeSmokeHeight = -1
const UpgradeIgnitionHeight = -2
const UpgradeLiftoffHeight = -3
var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
0: DrandMainnet,

6
build/params_shared_funcs.go
View File

@ -3,6 +3,8 @@ package build
import (
"sort"
"github.com/filecoin-project/go-address"
"github.com/libp2p/go-libp2p-core/protocol"
"github.com/filecoin-project/go-state-types/abi"
@ -44,3 +46,7 @@ func UseNewestNetwork() bool {
}
return false
}
func SetAddressNetwork(n address.Network) {
address.CurrentNetwork = n
}

18
build/params_shared_vals.go
View File

@ -4,6 +4,9 @@ package build
import (
"math/big"
"os"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/network"
@ -22,8 +25,8 @@ const UnixfsLinksPerLevel = 1024
// Consensus / Network
const AllowableClockDriftSecs = uint64(1)
const NewestNetworkVersion = network.Version2
const ActorUpgradeNetworkVersion = network.Version3
const NewestNetworkVersion = network.Version3
const ActorUpgradeNetworkVersion = network.Version4
// Epochs
const ForkLengthThreshold = Finality
@ -60,9 +63,16 @@ const TicketRandomnessLookback = abi.ChainEpoch(1)
const WinningPoStSectorSetLookback = abi.ChainEpoch(10)
// /////
// Address
const AddressMainnetEnvVar = "_mainnet_"
// /////
// Devnet settings
var Devnet = true
const FilBase = uint64(2_000_000_000)
const FilAllocStorageMining = uint64(1_100_000_000)
@ -75,6 +85,10 @@ var InitialRewardBalance *big.Int
func init() {
InitialRewardBalance = big.NewInt(int64(FilAllocStorageMining))
InitialRewardBalance = InitialRewardBalance.Mul(InitialRewardBalance, big.NewInt(int64(FilecoinPrecision)))
if os.Getenv("LOTUS_ADDRESS_TYPE") == AddressMainnetEnvVar {
SetAddressNetwork(address.Mainnet)
}
}
// Sync

6
build/params_testground.go
View File

@ -74,7 +74,9 @@ var (
UpgradeBreezeHeight abi.ChainEpoch = -1
BreezeGasTampingDuration abi.ChainEpoch = 0
UpgradeSmokeHeight abi.ChainEpoch = -1
UpgradeSmokeHeight abi.ChainEpoch = -1
UpgradeIgnitionHeight abi.ChainEpoch = -2
UpgradeLiftoffHeight abi.ChainEpoch = -3
DrandSchedule = map[abi.ChainEpoch]DrandEnum{
0: DrandMainnet,
@ -82,4 +84,6 @@ var (
NewestNetworkVersion = network.Version2
ActorUpgradeNetworkVersion = network.Version3
Devnet = true
)

8
build/params_testnet.go
View File

@ -21,12 +21,20 @@ const BreezeGasTampingDuration = 120
const UpgradeSmokeHeight = 51000
const UpgradeIgnitionHeight = 94000
// This signals our tentative epoch for mainnet launch. Can make it later, but not earlier.
// Miners, clients, developers, custodians all need time to prepare.
// We still have upgrades and state changes to do, but can happen after signaling timing here.
const UpgradeLiftoffHeight = 148888
func init() {
policy.SetConsensusMinerMinPower(abi.NewStoragePower(10 << 40))
policy.SetSupportedProofTypes(
abi.RegisteredSealProof_StackedDrg32GiBV1,
abi.RegisteredSealProof_StackedDrg64GiBV1,
)
Devnet = false
}
const BlockDelaySecs = uint64(builtin0.EpochDurationSeconds)

2
build/version.go
View File

@ -29,7 +29,7 @@ func buildType() string {
}
// BuildVersion is the local build version, set by build system
const BuildVersion = "0.7.2"
const BuildVersion = "0.8.0"
func UserVersion() string {
return BuildVersion + buildType() + CurrentCommit

2
chain/actors/builtin/builtin.go
View File

@ -21,7 +21,7 @@ const (
// Converts a network version into a specs-actors version.
func VersionForNetwork(version network.Version) Version {
switch version {
case network.Version0, network.Version1, network.Version2:
case network.Version0, network.Version1, network.Version2, network.Version3:
return Version0
default:
panic(fmt.Sprintf("unsupported network version %d", version))

4
chain/actors/builtin/miner/miner.go
View File

@ -20,6 +20,10 @@ import (
// Unchanged between v0 and v1 actors
var WPoStProvingPeriod = miner0.WPoStProvingPeriod
var WPoStPeriodDeadlines = miner0.WPoStPeriodDeadlines
var WPoStChallengeWindow = miner0.WPoStChallengeWindow
var WPoStChallengeLookback = miner0.WPoStChallengeLookback
var FaultDeclarationCutoff = miner0.FaultDeclarationCutoff
const MinSectorExpiration = miner0.MinSectorExpiration

7
chain/actors/builtin/miner/v0.go
View File

@ -6,6 +6,7 @@ import (
"github.com/libp2p/go-libp2p-core/peer"
cbg "github.com/whyrusleeping/cbor-gen"
"golang.org/x/xerrors"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-bitfield"
@ -96,9 +97,7 @@ func (s *state0) NumLiveSectors() (uint64, error) {
// GetSectorExpiration returns the effective expiration of the given sector.
//
// If the sector isn't found or has already been terminated, this method returns
// nil and no error. If the sector does not expire early, the Early expiration
// field is 0.
// If the sector does not expire early, the Early expiration field is 0.
func (s *state0) GetSectorExpiration(num abi.SectorNumber) (*SectorExpiration, error) {
dls, err := s.State.LoadDeadlines(s.store)
if err != nil {
@ -161,7 +160,7 @@ func (s *state0) GetSectorExpiration(num abi.SectorNumber) (*SectorExpiration, e
return nil, err
}
if out.Early == 0 && out.OnTime == 0 {
return nil, nil
return nil, xerrors.Errorf("failed to find sector %d", num)
}
return &out, nil
}

6
chain/actors/policy/policy.go
View File

@ -22,12 +22,10 @@ func SetSupportedProofTypes(types ...abi.RegisteredSealProof) {
// AddSupportedProofTypes sets supported proof types, across all actor versions.
// This should only be used for testing.
func AddSupportedProofTypes(types ...abi.RegisteredSealProof) {
newTypes := make(map[abi.RegisteredSealProof]struct{}, len(types))
for _, t := range types {
newTypes[t] = struct{}{}
// Set for all miner versions.
miner0.SupportedProofTypes[t] = struct{}{}
}
// Set for all miner versions.
miner0.SupportedProofTypes = newTypes
}
// SetPreCommitChallengeDelay sets the pre-commit challenge delay across all

36
chain/actors/policy/policy_test.go Normal file
View File

@ -0,0 +1,36 @@
package policy
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/filecoin-project/go-state-types/abi"
miner0 "github.com/filecoin-project/specs-actors/actors/builtin/miner"
)
func TestSupportedProofTypes(t *testing.T) {
var oldTypes []abi.RegisteredSealProof
for t := range miner0.SupportedProofTypes {
oldTypes = append(oldTypes, t)
}
t.Cleanup(func() {
SetSupportedProofTypes(oldTypes...)
})
SetSupportedProofTypes(abi.RegisteredSealProof_StackedDrg2KiBV1)
require.EqualValues(t,
miner0.SupportedProofTypes,
map[abi.RegisteredSealProof]struct{}{
abi.RegisteredSealProof_StackedDrg2KiBV1: {},
},
)
AddSupportedProofTypes(abi.RegisteredSealProof_StackedDrg8MiBV1)
require.EqualValues(t,
miner0.SupportedProofTypes,
map[abi.RegisteredSealProof]struct{}{
abi.RegisteredSealProof_StackedDrg2KiBV1: {},
abi.RegisteredSealProof_StackedDrg8MiBV1: {},
},
)
}

94
chain/messagepool/messagepool.go
View File

@ -75,8 +75,6 @@ var (
ErrRBFTooLowPremium = errors.New("replace by fee has too low GasPremium")
ErrTooManyPendingMessages = errors.New("too many pending messages for actor")
ErrNonceGap = errors.New("unfulfilled nonce gap")
ErrTryAgain = errors.New("state inconsistency while pushing message; please try again")
)
const (
@ -795,98 +793,6 @@ func (mp *MessagePool) getStateBalance(addr address.Address, ts *types.TipSet) (
return act.Balance, nil
}
func (mp *MessagePool) PushWithNonce(ctx context.Context, addr address.Address, cb func(address.Address, uint64) (*types.SignedMessage, error)) (*types.SignedMessage, error) {
// serialize push access to reduce lock contention
mp.addSema <- struct{}{}
defer func() {
<-mp.addSema
}()
mp.curTsLk.Lock()
mp.lk.Lock()
curTs := mp.curTs
fromKey := addr
if fromKey.Protocol() == address.ID {
var err error
fromKey, err = mp.api.StateAccountKey(ctx, fromKey, mp.curTs)
if err != nil {
mp.lk.Unlock()
mp.curTsLk.Unlock()
return nil, xerrors.Errorf("resolving sender key: %w", err)
}
}
nonce, err := mp.getNonceLocked(fromKey, mp.curTs)
if err != nil {
mp.lk.Unlock()
mp.curTsLk.Unlock()
return nil, xerrors.Errorf("get nonce locked failed: %w", err)
}
// release the locks for signing
mp.lk.Unlock()
mp.curTsLk.Unlock()
msg, err := cb(fromKey, nonce)
if err != nil {
return nil, err
}
err = mp.checkMessage(msg)
if err != nil {
return nil, err
}
msgb, err := msg.Serialize()
if err != nil {
return nil, err
}
// reacquire the locks and check state for consistency
mp.curTsLk.Lock()
defer mp.curTsLk.Unlock()
if mp.curTs != curTs {
return nil, ErrTryAgain
}
mp.lk.Lock()
defer mp.lk.Unlock()
nonce2, err := mp.getNonceLocked(fromKey, mp.curTs)
if err != nil {
return nil, xerrors.Errorf("get nonce locked failed: %w", err)
}
if nonce2 != nonce {
return nil, ErrTryAgain
}
publish, err := mp.verifyMsgBeforeAdd(msg, curTs, true)
if err != nil {
return nil, err
}
if err := mp.checkBalance(msg, curTs); err != nil {
return nil, err
}
if err := mp.addLocked(msg, false); err != nil {
return nil, xerrors.Errorf("add locked failed: %w", err)
}
if err := mp.addLocal(msg, msgb); err != nil {
log.Errorf("addLocal failed: %+v", err)
}
if publish {
err = mp.api.PubSubPublish(build.MessagesTopic(mp.netName), msgb)
}
return msg, err
}
func (mp *MessagePool) Remove(from address.Address, nonce uint64, applied bool) {
mp.lk.Lock()
defer mp.lk.Unlock()

124
chain/messagesigner/messagesigner.go Normal file
View File

@ -0,0 +1,124 @@
package messagesigner
import (
"bytes"
"context"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/lotus/chain/messagepool"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/wallet"
"github.com/filecoin-project/lotus/node/modules/dtypes"
"github.com/ipfs/go-datastore"
"github.com/ipfs/go-datastore/namespace"
logging "github.com/ipfs/go-log/v2"
cbg "github.com/whyrusleeping/cbor-gen"
"golang.org/x/xerrors"
)
const dsKeyActorNonce = "ActorNonce"
var log = logging.Logger("messagesigner")
type mpoolAPI interface {
GetNonce(address.Address) (uint64, error)
}
// MessageSigner keeps track of nonces per address, and increments the nonce
// when signing a message
type MessageSigner struct {
wallet *wallet.Wallet
mpool mpoolAPI
ds datastore.Batching
}
func NewMessageSigner(wallet *wallet.Wallet, mpool *messagepool.MessagePool, ds dtypes.MetadataDS) *MessageSigner {
return newMessageSigner(wallet, mpool, ds)
}
func newMessageSigner(wallet *wallet.Wallet, mpool mpoolAPI, ds dtypes.MetadataDS) *MessageSigner {
ds = namespace.Wrap(ds, datastore.NewKey("/message-signer/"))
return &MessageSigner{
wallet: wallet,
mpool: mpool,
ds: ds,
}
}
// SignMessage increments the nonce for the message From address, and signs
// the message
func (ms *MessageSigner) SignMessage(ctx context.Context, msg *types.Message) (*types.SignedMessage, error) {
nonce, err := ms.nextNonce(msg.From)
if err != nil {
return nil, xerrors.Errorf("failed to create nonce: %w", err)
}
msg.Nonce = nonce
sig, err := ms.wallet.Sign(ctx, msg.From, msg.Cid().Bytes())
if err != nil {
return nil, xerrors.Errorf("failed to sign message: %w", err)
}
return &types.SignedMessage{
Message: *msg,
Signature: *sig,
}, nil
}
// nextNonce increments the nonce.
// If there is no nonce in the datastore, gets the nonce from the message pool.
func (ms *MessageSigner) nextNonce(addr address.Address) (uint64, error) {
// Nonces used to be created by the mempool and we need to support nodes
// that have mempool nonces, so first check the mempool for a nonce for
// this address. Note that the mempool returns the actor state's nonce
// by default.
nonce, err := ms.mpool.GetNonce(addr)
if err != nil {
return 0, xerrors.Errorf("failed to get nonce from mempool: %w", err)
}
// Get the nonce for this address from the datastore
addrNonceKey := datastore.KeyWithNamespaces([]string{dsKeyActorNonce, addr.String()})
dsNonceBytes, err := ms.ds.Get(addrNonceKey)
switch {
case xerrors.Is(err, datastore.ErrNotFound):
// If a nonce for this address hasn't yet been created in the
// datastore, just use the nonce from the mempool
case err != nil:
return 0, xerrors.Errorf("failed to get nonce from datastore: %w", err)
default:
// There is a nonce in the datastore, so unmarshall and increment it
maj, val, err := cbg.CborReadHeader(bytes.NewReader(dsNonceBytes))
if err != nil {
return 0, xerrors.Errorf("failed to parse nonce from datastore: %w", err)
}
if maj != cbg.MajUnsignedInt {
return 0, xerrors.Errorf("bad cbor type parsing nonce from datastore")
}
dsNonce := val + 1
// The message pool nonce should be <= than the datastore nonce
if nonce <= dsNonce {
nonce = dsNonce
} else {
log.Warnf("mempool nonce was larger than datastore nonce (%d > %d)", nonce, dsNonce)
}
}
// Write the nonce for this address to the datastore
buf := bytes.Buffer{}
_, err = buf.Write(cbg.CborEncodeMajorType(cbg.MajUnsignedInt, nonce))
if err != nil {
return 0, xerrors.Errorf("failed to marshall nonce: %w", err)
}
err = ms.ds.Put(addrNonceKey, buf.Bytes())
if err != nil {
return 0, xerrors.Errorf("failed to write nonce to datastore: %w", err)
}
return nonce, nil
}

158
chain/messagesigner/messagesigner_test.go Normal file
View File

@ -0,0 +1,158 @@
package messagesigner
import (
"context"
"sync"
"testing"
"github.com/filecoin-project/lotus/chain/wallet"
"github.com/filecoin-project/go-state-types/crypto"
"github.com/stretchr/testify/require"
ds_sync "github.com/ipfs/go-datastore/sync"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/lotus/chain/types"
"github.com/ipfs/go-datastore"
)
type mockMpool struct {
lk sync.RWMutex
nonces map[address.Address]uint64
}
func newMockMpool() *mockMpool {
return &mockMpool{nonces: make(map[address.Address]uint64)}
}
func (mp *mockMpool) setNonce(addr address.Address, nonce uint64) {
mp.lk.Lock()
defer mp.lk.Unlock()
mp.nonces[addr] = nonce
}
func (mp *mockMpool) GetNonce(addr address.Address) (uint64, error) {
mp.lk.RLock()
defer mp.lk.RUnlock()
return mp.nonces[addr], nil
}
func TestMessageSignerSignMessage(t *testing.T) {
ctx := context.Background()
w, _ := wallet.NewWallet(wallet.NewMemKeyStore())
from1, err := w.GenerateKey(crypto.SigTypeSecp256k1)
require.NoError(t, err)
from2, err := w.GenerateKey(crypto.SigTypeSecp256k1)
require.NoError(t, err)
to1, err := w.GenerateKey(crypto.SigTypeSecp256k1)
require.NoError(t, err)
to2, err := w.GenerateKey(crypto.SigTypeSecp256k1)
require.NoError(t, err)
type msgSpec struct {
msg *types.Message
mpoolNonce [1]uint64
expNonce uint64
}
tests := []struct {
name string
msgs []msgSpec
}{{
// No nonce yet in datastore
name: "no nonce yet",
msgs: []msgSpec{{
msg: &types.Message{
To: to1,
From: from1,
},
expNonce: 0,
}},
}, {
// Get nonce value of zero from mpool
name: "mpool nonce zero",
msgs: []msgSpec{{
msg: &types.Message{
To: to1,
From: from1,
},
mpoolNonce: [1]uint64{0},
expNonce: 0,
}},
}, {
// Get non-zero nonce value from mpool
name: "mpool nonce set",
msgs: []msgSpec{{
msg: &types.Message{
To: to1,
From: from1,
},
mpoolNonce: [1]uint64{5},
expNonce: 5,
}, {
msg: &types.Message{
To: to1,
From: from1,
},
// Should adjust datastore nonce because mpool nonce is higher
mpoolNonce: [1]uint64{10},
expNonce: 10,
}},
}, {
// Nonce should increment independently for each address
name: "nonce increments per address",
msgs: []msgSpec{{
msg: &types.Message{
To: to1,
From: from1,
},
expNonce: 0,
}, {
msg: &types.Message{
To: to1,
From: from1,
},
expNonce: 1,
}, {
msg: &types.Message{
To: to2,
From: from2,
},
mpoolNonce: [1]uint64{5},
expNonce: 5,
}, {
msg: &types.Message{
To: to2,
From: from2,
},
expNonce: 6,
}, {
msg: &types.Message{
To: to1,
From: from1,
},
expNonce: 2,
}},
}}
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
mpool := newMockMpool()
ds := ds_sync.MutexWrap(datastore.NewMapDatastore())
ms := newMessageSigner(w, mpool, ds)
for _, m := range tt.msgs {
if len(m.mpoolNonce) == 1 {
mpool.setNonce(m.msg.From, m.mpoolNonce[0])
}
smsg, err := ms.SignMessage(ctx, m.msg)
require.NoError(t, err)
require.Equal(t, m.expNonce, smsg.Message.Nonce)
}
})
}
}

381
chain/stmgr/forks.go
View File

@ -1,44 +1,59 @@
package stmgr
import (
"bytes"
"context"
"encoding/binary"
"math"
multisig0 "github.com/filecoin-project/specs-actors/actors/builtin/multisig"
"github.com/filecoin-project/lotus/chain/actors/builtin/multisig"
"github.com/filecoin-project/lotus/chain/state"
"github.com/filecoin-project/specs-actors/actors/migration/nv3"
"github.com/ipfs/go-cid"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/big"
builtin0 "github.com/filecoin-project/specs-actors/actors/builtin"
init0 "github.com/filecoin-project/specs-actors/actors/builtin/init"
miner0 "github.com/filecoin-project/specs-actors/actors/builtin/miner"
power0 "github.com/filecoin-project/specs-actors/actors/builtin/power"
adt0 "github.com/filecoin-project/specs-actors/actors/util/adt"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/vm"
cbor "github.com/ipfs/go-ipld-cbor"
"golang.org/x/xerrors"
)
var ForksAtHeight = map[abi.ChainEpoch]func(context.Context, *StateManager, types.StateTree, *types.TipSet) error{
build.UpgradeBreezeHeight: UpgradeFaucetBurnRecovery,
var ForksAtHeight = map[abi.ChainEpoch]func(context.Context, *StateManager, ExecCallback, cid.Cid, *types.TipSet) (cid.Cid, error){
build.UpgradeBreezeHeight: UpgradeFaucetBurnRecovery,
build.UpgradeIgnitionHeight: UpgradeIgnition,
build.UpgradeLiftoffHeight: UpgradeLiftoff,
}
func (sm *StateManager) handleStateForks(ctx context.Context, st types.StateTree, height abi.ChainEpoch, ts *types.TipSet) (err error) {
func (sm *StateManager) handleStateForks(ctx context.Context, root cid.Cid, height abi.ChainEpoch, cb ExecCallback, ts *types.TipSet) (cid.Cid, error) {
retCid := root
var err error
f, ok := ForksAtHeight[height]
if ok {
err := f(ctx, sm, st, ts)
retCid, err = f(ctx, sm, cb, root, ts)
if err != nil {
return err
return cid.Undef, err
}
}
return nil
return retCid, nil
}
type forEachTree interface {
ForEach(func(address.Address, *types.Actor) error) error
}
func doTransfer(tree types.StateTree, from, to address.Address, amt abi.TokenAmount) error {
func doTransfer(cb ExecCallback, tree types.StateTree, from, to address.Address, amt abi.TokenAmount) error {
fromAct, err := tree.GetActor(from)
if err != nil {
return xerrors.Errorf("failed to get 'from' actor for transfer: %w", err)
@ -64,10 +79,43 @@ func doTransfer(tree types.StateTree, from, to address.Address, amt abi.TokenAmo
return xerrors.Errorf("failed to persist to actor: %w", err)
}
if cb != nil {
// record the transfer in execution traces
fakeMsg := &types.Message{
From: from,
To: to,
Value: amt,
Nonce: math.MaxUint64,
}
fakeRct := &types.MessageReceipt{
ExitCode: 0,
Return: nil,
GasUsed: 0,
}
if err := cb(fakeMsg.Cid(), fakeMsg, &vm.ApplyRet{
MessageReceipt: *fakeRct,
ActorErr: nil,
ExecutionTrace: types.ExecutionTrace{
Msg: fakeMsg,
MsgRct: fakeRct,
Error: "",
Duration: 0,
GasCharges: nil,
Subcalls: nil,
},
Duration: 0,
GasCosts: vm.ZeroGasOutputs(),
}); err != nil {
return xerrors.Errorf("recording transfer: %w", err)
}
}
return nil
}
func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types.StateTree, ts *types.TipSet) error {
func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, cb ExecCallback, root cid.Cid, ts *types.TipSet) (cid.Cid, error) {
// Some initial parameters
FundsForMiners := types.FromFil(1_000_000)
LookbackEpoch := abi.ChainEpoch(32000)
@ -94,22 +142,22 @@ func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types
// Grab lookback state for account checks
lbts, err := sm.ChainStore().GetTipsetByHeight(ctx, LookbackEpoch, ts, false)
if err != nil {
return xerrors.Errorf("failed to get tipset at lookback height: %w", err)
return cid.Undef, xerrors.Errorf("failed to get tipset at lookback height: %w", err)
}
lbtree, err := sm.ParentState(lbts)
if err != nil {
return xerrors.Errorf("loading state tree failed: %w", err)
return cid.Undef, xerrors.Errorf("loading state tree failed: %w", err)
}
ReserveAddress, err := address.NewFromString("t090")
if err != nil {
return xerrors.Errorf("failed to parse reserve address: %w", err)
return cid.Undef, xerrors.Errorf("failed to parse reserve address: %w", err)
}
fetree, ok := tree.(forEachTree)
if !ok {
return xerrors.Errorf("fork transition state tree doesnt support ForEach (%T)", tree)
tree, err := sm.StateTree(root)
if err != nil {
return cid.Undef, xerrors.Errorf("getting state tree: %w", err)
}
type transfer struct {
@ -121,7 +169,7 @@ func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types
var transfers []transfer
// Take all excess funds away, put them into the reserve account
err = fetree.ForEach(func(addr address.Address, act *types.Actor) error {
err = tree.ForEach(func(addr address.Address, act *types.Actor) error {
switch act.Code {
case builtin0.AccountActorCodeID, builtin0.MultisigActorCodeID, builtin0.PaymentChannelActorCodeID:
sysAcc, err := isSystemAccount(addr)
@ -163,13 +211,13 @@ func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types
return nil
})
if err != nil {
return xerrors.Errorf("foreach over state tree failed: %w", err)
return cid.Undef, xerrors.Errorf("foreach over state tree failed: %w", err)
}
// Execute transfers from previous step
for _, t := range transfers {
if err := doTransfer(tree, t.From, t.To, t.Amt); err != nil {
return xerrors.Errorf("transfer %s %s->%s failed: %w", t.Amt, t.From, t.To, err)
if err := doTransfer(cb, tree, t.From, t.To, t.Amt); err != nil {
return cid.Undef, xerrors.Errorf("transfer %s %s->%s failed: %w", t.Amt, t.From, t.To, err)
}
}
@ -177,19 +225,19 @@ func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types
var ps power0.State
powAct, err := tree.GetActor(builtin0.StoragePowerActorAddr)
if err != nil {
return xerrors.Errorf("failed to load power actor: %w", err)
return cid.Undef, xerrors.Errorf("failed to load power actor: %w", err)
}
cst := cbor.NewCborStore(sm.ChainStore().Blockstore())
if err := cst.Get(ctx, powAct.Head, &ps); err != nil {
return xerrors.Errorf("failed to get power actor state: %w", err)
return cid.Undef, xerrors.Errorf("failed to get power actor state: %w", err)
}
totalPower := ps.TotalBytesCommitted
var transfersBack []transfer
// Now, we return some funds to places where they are needed
err = fetree.ForEach(func(addr address.Address, act *types.Actor) error {
err = tree.ForEach(func(addr address.Address, act *types.Actor) error {
lbact, err := lbtree.GetActor(addr)
if err != nil {
if !xerrors.Is(err, types.ErrActorNotFound) {
@ -267,53 +315,310 @@ func UpgradeFaucetBurnRecovery(ctx context.Context, sm *StateManager, tree types
return nil
})
if err != nil {
return xerrors.Errorf("foreach over state tree failed: %w", err)
return cid.Undef, xerrors.Errorf("foreach over state tree failed: %w", err)
}
for _, t := range transfersBack {
if err := doTransfer(tree, t.From, t.To, t.Amt); err != nil {
return xerrors.Errorf("transfer %s %s->%s failed: %w", t.Amt, t.From, t.To, err)
if err := doTransfer(cb, tree, t.From, t.To, t.Amt); err != nil {
return cid.Undef, xerrors.Errorf("transfer %s %s->%s failed: %w", t.Amt, t.From, t.To, err)
}
}
// transfer all burnt funds back to the reserve account
burntAct, err := tree.GetActor(builtin0.BurntFundsActorAddr)
if err != nil {
return xerrors.Errorf("failed to load burnt funds actor: %w", err)
return cid.Undef, xerrors.Errorf("failed to load burnt funds actor: %w", err)
}
if err := doTransfer(tree, builtin0.BurntFundsActorAddr, ReserveAddress, burntAct.Balance); err != nil {
return xerrors.Errorf("failed to unburn funds: %w", err)
if err := doTransfer(cb, tree, builtin0.BurntFundsActorAddr, ReserveAddress, burntAct.Balance); err != nil {
return cid.Undef, xerrors.Errorf("failed to unburn funds: %w", err)
}
// Top up the reimbursement service
reimbAddr, err := address.NewFromString("t0111")
if err != nil {
return xerrors.Errorf("failed to parse reimbursement service address")
return cid.Undef, xerrors.Errorf("failed to parse reimbursement service address")
}
reimb, err := tree.GetActor(reimbAddr)
if err != nil {
return xerrors.Errorf("failed to load reimbursement account actor: %w", err)
return cid.Undef, xerrors.Errorf("failed to load reimbursement account actor: %w", err)
}
difference := types.BigSub(DesiredReimbursementBalance, reimb.Balance)
if err := doTransfer(tree, ReserveAddress, reimbAddr, difference); err != nil {
return xerrors.Errorf("failed to top up reimbursement account: %w", err)
if err := doTransfer(cb, tree, ReserveAddress, reimbAddr, difference); err != nil {
return cid.Undef, xerrors.Errorf("failed to top up reimbursement account: %w", err)
}
// Now, a final sanity check to make sure the balances all check out
total := abi.NewTokenAmount(0)
err = fetree.ForEach(func(addr address.Address, act *types.Actor) error {
err = tree.ForEach(func(addr address.Address, act *types.Actor) error {
total = types.BigAdd(total, act.Balance)
return nil
})
if err != nil {
return xerrors.Errorf("checking final state balance failed: %w", err)
return cid.Undef, xerrors.Errorf("checking final state balance failed: %w", err)
}
exp := types.FromFil(build.FilBase)
if !exp.Equals(total) {
return xerrors.Errorf("resultant state tree account balance was not correct: %s", total)
return cid.Undef, xerrors.Errorf("resultant state tree account balance was not correct: %s", total)
}
return tree.Flush(ctx)
}
func UpgradeIgnition(ctx context.Context, sm *StateManager, cb ExecCallback, root cid.Cid, ts *types.TipSet) (cid.Cid, error) {
store := sm.cs.Store(ctx)
nst, err := nv3.MigrateStateTree(ctx, store, root, build.UpgradeIgnitionHeight)
if err != nil {
return cid.Undef, xerrors.Errorf("migrating actors state: %w", err)
}
tree, err := sm.StateTree(nst)
if err != nil {
return cid.Undef, xerrors.Errorf("getting state tree: %w", err)
}
err = setNetworkName(ctx, store, tree, "ignition")
if err != nil {
return cid.Undef, xerrors.Errorf("setting network name: %w", err)
}
split1, err := address.NewFromString("t0115")
if err != nil {
return cid.Undef, xerrors.Errorf("first split address: %w", err)
}
split2, err := address.NewFromString("t0116")
if err != nil {
return cid.Undef, xerrors.Errorf("second split address: %w", err)
}
err = resetGenesisMsigs(ctx, sm, store, tree)
if err != nil {
return cid.Undef, xerrors.Errorf("resetting genesis msig start epochs: %w", err)
}
err = splitGenesisMultisig(ctx, cb, split1, store, tree, 50)
if err != nil {
return cid.Undef, xerrors.Errorf("splitting first msig: %w", err)
}
err = splitGenesisMultisig(ctx, cb, split2, store, tree, 50)
if err != nil {
return cid.Undef, xerrors.Errorf("splitting second msig: %w", err)
}
err = nv3.CheckStateTree(ctx, store, nst, build.UpgradeIgnitionHeight, builtin0.TotalFilecoin)
if err != nil {
return cid.Undef, xerrors.Errorf("sanity check after ignition upgrade failed: %w", err)
}
return tree.Flush(ctx)
}
func UpgradeLiftoff(ctx context.Context, sm *StateManager, cb ExecCallback, root cid.Cid, ts *types.TipSet) (cid.Cid, error) {
tree, err := sm.StateTree(root)
if err != nil {
return cid.Undef, xerrors.Errorf("getting state tree: %w", err)
}
err = setNetworkName(ctx, sm.cs.Store(ctx), tree, "mainnet")
if err != nil {
return cid.Undef, xerrors.Errorf("setting network name: %w", err)
}
return tree.Flush(ctx)
}
func setNetworkName(ctx context.Context, store adt0.Store, tree *state.StateTree, name string) error {
ia, err := tree.GetActor(builtin0.InitActorAddr)
if err != nil {
return xerrors.Errorf("getting init actor: %w", err)
}
var initState init0.State
if err := store.Get(ctx, ia.Head, &initState); err != nil {
return xerrors.Errorf("reading init state: %w", err)
}
initState.NetworkName = name
ia.Head, err = store.Put(ctx, &initState)
if err != nil {
return xerrors.Errorf("writing new init state: %w", err)
}
if err := tree.SetActor(builtin0.InitActorAddr, ia); err != nil {
return xerrors.Errorf("setting init actor: %w", err)
}
return nil
}
func splitGenesisMultisig(ctx context.Context, cb ExecCallback, addr address.Address, store adt0.Store, tree *state.StateTree, portions uint64) error {
if portions < 1 {
return xerrors.Errorf("cannot split into 0 portions")
}
mact, err := tree.GetActor(addr)
if err != nil {
return xerrors.Errorf("getting msig actor: %w", err)
}
mst, err := multisig.Load(store, mact)
if err != nil {
return xerrors.Errorf("getting msig state: %w", err)
}
signers, err := mst.Signers()
if err != nil {
return xerrors.Errorf("getting msig signers: %w", err)
}
thresh, err := mst.Threshold()
if err != nil {
return xerrors.Errorf("getting msig threshold: %w", err)
}
ibal, err := mst.InitialBalance()
if err != nil {
return xerrors.Errorf("getting msig initial balance: %w", err)
}
se, err := mst.StartEpoch()
if err != nil {
return xerrors.Errorf("getting msig start epoch: %w", err)
}
ud, err := mst.UnlockDuration()
if err != nil {
return xerrors.Errorf("getting msig unlock duration: %w", err)
}
pending, err := adt0.MakeEmptyMap(store).Root()
if err != nil {
return xerrors.Errorf("failed to create empty map: %w", err)
}
newIbal := big.Div(ibal, types.NewInt(portions))
newState := &multisig0.State{
Signers: signers,
NumApprovalsThreshold: thresh,
NextTxnID: 0,
InitialBalance: newIbal,
StartEpoch: se,
UnlockDuration: ud,
PendingTxns: pending,
}
scid, err := store.Put(ctx, newState)
if err != nil {
return xerrors.Errorf("storing new state: %w", err)
}
newActor := types.Actor{
Code: builtin0.MultisigActorCodeID,
Head: scid,
Nonce: 0,
Balance: big.Zero(),
}
i := uint64(0)
for i < portions {
keyAddr, err := makeKeyAddr(addr, i)
if err != nil {
return xerrors.Errorf("creating key address: %w", err)
}
idAddr, err := tree.RegisterNewAddress(keyAddr)
if err != nil {
return xerrors.Errorf("registering new address: %w", err)
}
err = tree.SetActor(idAddr, &newActor)
if err != nil {
return xerrors.Errorf("setting new msig actor state: %w", err)
}
if err := doTransfer(cb, tree, addr, idAddr, newIbal); err != nil {
return xerrors.Errorf("transferring split msig balance: %w", err)
}
i++
}
return nil
}
func makeKeyAddr(splitAddr address.Address, count uint64) (address.Address, error) {
var b bytes.Buffer
if err := splitAddr.MarshalCBOR(&b); err != nil {
return address.Undef, xerrors.Errorf("marshalling split address: %w", err)
}
if err := binary.Write(&b, binary.BigEndian, count); err != nil {
return address.Undef, xerrors.Errorf("writing count into a buffer: %w", err)
}
if err := binary.Write(&b, binary.BigEndian, []byte("Ignition upgrade")); err != nil {
return address.Undef, xerrors.Errorf("writing fork name into a buffer: %w", err)
}
addr, err := address.NewActorAddress(b.Bytes())
if err != nil {
return address.Undef, xerrors.Errorf("create actor address: %w", err)
}
return addr, nil
}
func resetGenesisMsigs(ctx context.Context, sm *StateManager, store adt0.Store, tree *state.StateTree) error {
gb, err := sm.cs.GetGenesis()
if err != nil {
return xerrors.Errorf("getting genesis block: %w", err)
}
gts, err := types.NewTipSet([]*types.BlockHeader{gb})
if err != nil {
return xerrors.Errorf("getting genesis tipset: %w", err)
}
cst := cbor.NewCborStore(sm.cs.Blockstore())
genesisTree, err := state.LoadStateTree(cst, gts.ParentState())
if err != nil {
return xerrors.Errorf("loading state tree: %w", err)
}
err = genesisTree.ForEach(func(addr address.Address, genesisActor *types.Actor) error {
if genesisActor.Code == builtin0.MultisigActorCodeID {
currActor, err := tree.GetActor(addr)
if err != nil {
return xerrors.Errorf("loading actor: %w", err)
}
var currState multisig0.State
if err := store.Get(ctx, currActor.Head, &currState); err != nil {
return xerrors.Errorf("reading multisig state: %w", err)
}
currState.StartEpoch = build.UpgradeLiftoffHeight
currActor.Head, err = store.Put(ctx, &currState)
if err != nil {
return xerrors.Errorf("writing new multisig state: %w", err)
}
if err := tree.SetActor(addr, currActor); err != nil {
return xerrors.Errorf("setting multisig actor: %w", err)
}
}
return nil
})
if err != nil {
return xerrors.Errorf("iterating over genesis actors: %w", err)
}
return nil

18
chain/stmgr/forks_test.go
View File

@ -25,6 +25,7 @@ import (
_ "github.com/filecoin-project/lotus/lib/sigs/bls"
_ "github.com/filecoin-project/lotus/lib/sigs/secp"
"github.com/ipfs/go-cid"
cbor "github.com/ipfs/go-ipld-cbor"
logging "github.com/ipfs/go-log"
cbg "github.com/whyrusleeping/cbor-gen"
@ -114,33 +115,38 @@ func TestForkHeightTriggers(t *testing.T) {
t.Fatal(err)
}
stmgr.ForksAtHeight[testForkHeight] = func(ctx context.Context, sm *StateManager, st types.StateTree, ts *types.TipSet) error {
stmgr.ForksAtHeight[testForkHeight] = func(ctx context.Context, sm *StateManager, cb ExecCallback, root cid.Cid, ts *types.TipSet) (cid.Cid, error) {
cst := cbor.NewCborStore(sm.ChainStore().Blockstore())
st, err := sm.StateTree(root)
if err != nil {
return cid.Undef, xerrors.Errorf("getting state tree: %w", err)
}
act, err := st.GetActor(taddr)
if err != nil {
return err
return cid.Undef, err
}
var tas testActorState
if err := cst.Get(ctx, act.Head, &tas); err != nil {
return xerrors.Errorf("in fork handler, failed to run get: %w", err)
return cid.Undef, xerrors.Errorf("in fork handler, failed to run get: %w", err)
}
tas.HasUpgraded = 55
ns, err := cst.Put(ctx, &tas)
if err != nil {
return err
return cid.Undef, err
}
act.Head = ns
if err := st.SetActor(taddr, act); err != nil {
return err
return cid.Undef, err
}
return nil
return st.Flush(ctx)
}
inv.Register(builtin.PaymentChannelActorCodeID, &testActor{}, &testActorState{})

194
chain/stmgr/stmgr.go
View File

@ -41,12 +41,13 @@ var log = logging.Logger("statemgr")
type StateManager struct {
cs *store.ChainStore
stCache map[string][]cid.Cid
compWait map[string]chan struct{}
stlk sync.Mutex
genesisMsigLk sync.Mutex
newVM func(context.Context, *vm.VMOpts) (*vm.VM, error)
genInfo *genesisInfo
stCache map[string][]cid.Cid
compWait map[string]chan struct{}
stlk sync.Mutex
genesisMsigLk sync.Mutex
newVM func(context.Context, *vm.VMOpts) (*vm.VM, error)
preIgnitionGenInfos *genesisInfo
postIgnitionGenInfos *genesisInfo
}
func NewStateManager(cs *store.ChainStore) *StateManager {
@ -123,9 +124,8 @@ func (sm *StateManager) TipSetState(ctx context.Context, ts *types.TipSet) (st c
return st, rec, nil
}
func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (cid.Cid, []*api.InvocResult, error) {
var trace []*api.InvocResult
st, _, err := sm.computeTipSetState(ctx, ts, func(mcid cid.Cid, msg *types.Message, ret *vm.ApplyRet) error {
func traceFunc(trace *[]*api.InvocResult) func(mcid cid.Cid, msg *types.Message, ret *vm.ApplyRet) error {
return func(mcid cid.Cid, msg *types.Message, ret *vm.ApplyRet) error {
ir := &api.InvocResult{
Msg: msg,
MsgRct: &ret.MessageReceipt,
@ -135,9 +135,14 @@ func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (c
if ret.ActorErr != nil {
ir.Error = ret.ActorErr.Error()
}
trace = append(trace, ir)
*trace = append(*trace, ir)
return nil
})
}
}
func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (cid.Cid, []*api.InvocResult, error) {
var trace []*api.InvocResult
st, _, err := sm.computeTipSetState(ctx, ts, traceFunc(&trace))
if err != nil {
return cid.Undef, nil, err
}
@ -149,20 +154,24 @@ type ExecCallback func(cid.Cid, *types.Message, *vm.ApplyRet) error
func (sm *StateManager) ApplyBlocks(ctx context.Context, parentEpoch abi.ChainEpoch, pstate cid.Cid, bms []store.BlockMessages, epoch abi.ChainEpoch, r vm.Rand, cb ExecCallback, baseFee abi.TokenAmount, ts *types.TipSet) (cid.Cid, cid.Cid, error) {
vmopt := &vm.VMOpts{
StateBase: pstate,
Epoch: epoch,
Rand: r,
Bstore: sm.cs.Blockstore(),
Syscalls: sm.cs.VMSys(),
CircSupplyCalc: sm.GetCirculatingSupply,
NtwkVersion: sm.GetNtwkVersion,
BaseFee: baseFee,
makeVmWithBaseState := func(base cid.Cid) (*vm.VM, error) {
vmopt := &vm.VMOpts{
StateBase: base,
Epoch: epoch,
Rand: r,
Bstore: sm.cs.Blockstore(),
Syscalls: sm.cs.VMSys(),
CircSupplyCalc: sm.GetCirculatingSupply,
NtwkVersion: sm.GetNtwkVersion,
BaseFee: baseFee,
}
return sm.newVM(ctx, vmopt)
}
vmi, err := sm.newVM(ctx, vmopt)
vmi, err := makeVmWithBaseState(pstate)
if err != nil {
return cid.Undef, cid.Undef, xerrors.Errorf("instantiating VM failed: %w", err)
return cid.Undef, cid.Undef, xerrors.Errorf("making vm: %w", err)
}
runCron := func() error {
@ -202,19 +211,32 @@ func (sm *StateManager) ApplyBlocks(ctx context.Context, parentEpoch abi.ChainEp
for i := parentEpoch; i < epoch; i++ {
// handle state forks
// XXX: The state tree
err = sm.handleStateForks(ctx, vmi.StateTree(), i, ts)
newState, err := sm.handleStateForks(ctx, pstate, i, cb, ts)
if err != nil {
return cid.Undef, cid.Undef, xerrors.Errorf("error handling state forks: %w", err)
}
if pstate != newState {
vmi, err = makeVmWithBaseState(newState)
if err != nil {
return cid.Undef, cid.Undef, xerrors.Errorf("making vm: %w", err)
}
}
if i > parentEpoch {
// run cron for null rounds if any
if err := runCron(); err != nil {
return cid.Cid{}, cid.Cid{}, err
}
newState, err = vmi.Flush(ctx)
if err != nil {
return cid.Undef, cid.Undef, xerrors.Errorf("flushing vm: %w", err)
}
}
vmi.SetBlockHeight(i + 1)
pstate = newState
}
var receipts []cbg.CBORMarshaler
@ -904,7 +926,7 @@ func (sm *StateManager) setupGenesisActors(ctx context.Context) error {
gi.genesisMsigs = append(gi.genesisMsigs, ns)
}
sm.genInfo = &gi
sm.preIgnitionGenInfos = &gi
return nil
}
@ -912,7 +934,7 @@ func (sm *StateManager) setupGenesisActors(ctx context.Context) error {
// sets up information about the actors in the genesis state
// For testnet we use a hardcoded set of multisig states, instead of what's actually in the genesis multisigs
// We also do not consider ANY account actors (including the faucet)
func (sm *StateManager) setupGenesisActorsTestnet(ctx context.Context) error {
func (sm *StateManager) setupPreIgnitionGenesisActorsTestnet(ctx context.Context) error {
gi := genesisInfo{}
@ -981,7 +1003,87 @@ func (sm *StateManager) setupGenesisActorsTestnet(ctx context.Context) error {
gi.genesisMsigs = append(gi.genesisMsigs, ns)
}
sm.genInfo = &gi
sm.preIgnitionGenInfos = &gi
return nil
}
// sets up information about the actors in the genesis state, post the ignition fork
func (sm *StateManager) setupPostIgnitionGenesisActors(ctx context.Context) error {
gi := genesisInfo{}
gb, err := sm.cs.GetGenesis()
if err != nil {
return xerrors.Errorf("getting genesis block: %w", err)
}
gts, err := types.NewTipSet([]*types.BlockHeader{gb})
if err != nil {
return xerrors.Errorf("getting genesis tipset: %w", err)
}
st, _, err := sm.TipSetState(ctx, gts)
if err != nil {
return xerrors.Errorf("getting genesis tipset state: %w", err)
}
cst := cbor.NewCborStore(sm.cs.Blockstore())
sTree, err := state.LoadStateTree(cst, st)
if err != nil {
return xerrors.Errorf("loading state tree: %w", err)
}
// Unnecessary, should be removed
gi.genesisMarketFunds, err = getFilMarketLocked(ctx, sTree)
if err != nil {
return xerrors.Errorf("setting up genesis market funds: %w", err)
}
// Unnecessary, should be removed
gi.genesisPledge, err = getFilPowerLocked(ctx, sTree)
if err != nil {
return xerrors.Errorf("setting up genesis pledge: %w", err)
}
totalsByEpoch := make(map[abi.ChainEpoch]abi.TokenAmount)
// 6 months
sixMonths := abi.ChainEpoch(183 * builtin0.EpochsInDay)
totalsByEpoch[sixMonths] = big.NewInt(49_929_341)
totalsByEpoch[sixMonths] = big.Add(totalsByEpoch[sixMonths], big.NewInt(32_787_700))
// 1 year
oneYear := abi.ChainEpoch(365 * builtin0.EpochsInDay)
totalsByEpoch[oneYear] = big.NewInt(22_421_712)
// 2 years
twoYears := abi.ChainEpoch(2 * 365 * builtin0.EpochsInDay)
totalsByEpoch[twoYears] = big.NewInt(7_223_364)
// 3 years
threeYears := abi.ChainEpoch(3 * 365 * builtin0.EpochsInDay)
totalsByEpoch[threeYears] = big.NewInt(87_637_883)
// 6 years
sixYears := abi.ChainEpoch(6 * 365 * builtin0.EpochsInDay)
totalsByEpoch[sixYears] = big.NewInt(100_000_000)
totalsByEpoch[sixYears] = big.Add(totalsByEpoch[sixYears], big.NewInt(300_000_000))
gi.genesisMsigs = make([]msig0.State, 0, len(totalsByEpoch))
for k, v := range totalsByEpoch {
ns := msig0.State{
// In the pre-ignition logic, we incorrectly set this value in Fil, not attoFil, an off-by-10^18 error
InitialBalance: big.Mul(v, big.NewInt(int64(build.FilecoinPrecision))),
UnlockDuration: k,
PendingTxns: cid.Undef,
// In the pre-ignition logic, the start epoch was 0. This changes in the fork logic of the Ignition upgrade itself.
StartEpoch: build.UpgradeLiftoffHeight,
}
gi.genesisMsigs = append(gi.genesisMsigs, ns)
}
sm.postIgnitionGenInfos = &gi
return nil
}
@ -991,13 +1093,23 @@ func (sm *StateManager) setupGenesisActorsTestnet(ctx context.Context) error {
// - For Accounts, it counts max(currentBalance - genesisBalance, 0).
func (sm *StateManager) GetFilVested(ctx context.Context, height abi.ChainEpoch, st *state.StateTree) (abi.TokenAmount, error) {
vf := big.Zero()
for _, v := range sm.genInfo.genesisMsigs {
au := big.Sub(v.InitialBalance, v.AmountLocked(height))
vf = big.Add(vf, au)
if height <= build.UpgradeIgnitionHeight {
for _, v := range sm.preIgnitionGenInfos.genesisMsigs {
au := big.Sub(v.InitialBalance, v.AmountLocked(height))
vf = big.Add(vf, au)
}
} else {
for _, v := range sm.postIgnitionGenInfos.genesisMsigs {
// In the pre-ignition logic, we simply called AmountLocked(height), assuming startEpoch was 0.
// The start epoch changed in the Ignition upgrade.
au := big.Sub(v.InitialBalance, v.AmountLocked(height-v.StartEpoch))
vf = big.Add(vf, au)
}
}
// there should not be any such accounts in testnet (and also none in mainnet?)
for _, v := range sm.genInfo.genesisActors {
// continue to use preIgnitionGenInfos, nothing changed at the Ignition epoch
for _, v := range sm.preIgnitionGenInfos.genesisActors {
act, err := st.GetActor(v.addr)
if err != nil {
return big.Zero(), xerrors.Errorf("failed to get actor: %w", err)
@ -1009,8 +1121,10 @@ func (sm *StateManager) GetFilVested(ctx context.Context, height abi.ChainEpoch,
}
}
vf = big.Add(vf, sm.genInfo.genesisPledge)
vf = big.Add(vf, sm.genInfo.genesisMarketFunds)
// continue to use preIgnitionGenInfos, nothing changed at the Ignition epoch
vf = big.Add(vf, sm.preIgnitionGenInfos.genesisPledge)
// continue to use preIgnitionGenInfos, nothing changed at the Ignition epoch
vf = big.Add(vf, sm.preIgnitionGenInfos.genesisMarketFunds)
return vf, nil
}
@ -1084,10 +1198,16 @@ func GetFilBurnt(ctx context.Context, st *state.StateTree) (abi.TokenAmount, err
func (sm *StateManager) GetCirculatingSupplyDetailed(ctx context.Context, height abi.ChainEpoch, st *state.StateTree) (api.CirculatingSupply, error) {
sm.genesisMsigLk.Lock()
defer sm.genesisMsigLk.Unlock()
if sm.genInfo == nil {
err := sm.setupGenesisActorsTestnet(ctx)
if sm.preIgnitionGenInfos == nil {
err := sm.setupPreIgnitionGenesisActorsTestnet(ctx)
if err != nil {
return api.CirculatingSupply{}, xerrors.Errorf("failed to setup genesis information: %w", err)
return api.CirculatingSupply{}, xerrors.Errorf("failed to setup pre-ignition genesis information: %w", err)
}
}
if sm.postIgnitionGenInfos == nil {
err := sm.setupPostIgnitionGenesisActors(ctx)
if err != nil {
return api.CirculatingSupply{}, xerrors.Errorf("failed to setup post-ignition genesis information: %w", err)
}
}
@ -1152,6 +1272,10 @@ func (sm *StateManager) GetNtwkVersion(ctx context.Context, height abi.ChainEpoc
return network.Version1
}
if height <= build.UpgradeIgnitionHeight {
return network.Version2
}
return build.NewestNetworkVersion
}

29
chain/stmgr/utils.go
View File

@ -102,6 +102,7 @@ func GetPowerRaw(ctx context.Context, sm *StateManager, st cid.Cid, maddr addres
}
var mpow power.Claim
var minpow bool
if maddr != address.Undef {
var found bool
mpow, found, err = pas.MinerPower(maddr)
@ -109,11 +110,11 @@ func GetPowerRaw(ctx context.Context, sm *StateManager, st cid.Cid, maddr addres
// TODO: return an error when not found?
return power.Claim{}, power.Claim{}, false, err
}
}
minpow, err := pas.MinerNominalPowerMeetsConsensusMinimum(maddr)
if err != nil {
return power.Claim{}, power.Claim{}, false, err
minpow, err = pas.MinerNominalPowerMeetsConsensusMinimum(maddr)
if err != nil {
return power.Claim{}, power.Claim{}, false, err
}
}
return mpow, tpow, minpow, nil
@ -368,6 +369,16 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
return cid.Undef, nil, err
}
for i := ts.Height(); i < height; i++ {
// handle state forks
base, err = sm.handleStateForks(ctx, base, i, traceFunc(&trace), ts)
if err != nil {
return cid.Undef, nil, xerrors.Errorf("error handling state forks: %w", err)
}
// TODO: should we also run cron here?
}
r := store.NewChainRand(sm.cs, ts.Cids())
vmopt := &vm.VMOpts{
StateBase: base,
@ -384,16 +395,6 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
return cid.Undef, nil, err
}
for i := ts.Height(); i < height; i++ {
// handle state forks
err = sm.handleStateForks(ctx, vmi.StateTree(), i, ts)
if err != nil {
return cid.Undef, nil, xerrors.Errorf("error handling state forks: %w", err)
}
// TODO: should we also run cron here?
}
for i, msg := range msgs {
// TODO: Use the signed message length for secp messages
ret, err := vmi.ApplyMessage(ctx, msg)

52
chain/store/store.go
View File

@ -10,14 +10,18 @@ import (
"strconv"
"sync"
"golang.org/x/sync/errgroup"
"github.com/filecoin-project/go-state-types/crypto"
"github.com/minio/blake2b-simd"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/specs-actors/actors/builtin"
"github.com/filecoin-project/specs-actors/actors/util/adt"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/vm"
"github.com/filecoin-project/lotus/journal"
bstore "github.com/filecoin-project/lotus/lib/blockstore"
@ -282,6 +286,16 @@ func (cs *ChainStore) MarkBlockAsValidated(ctx context.Context, blkid cid.Cid) e
return nil
}
func (cs *ChainStore) UnmarkBlockAsValidated(ctx context.Context, blkid cid.Cid) error {
key := blockValidationCacheKeyPrefix.Instance(blkid.String())
if err := cs.ds.Delete(key); err != nil {
return xerrors.Errorf("removing from valid block cache: %w", err)
}
return nil
}
func (cs *ChainStore) SetGenesis(b *types.BlockHeader) error {
ts, err := types.NewTipSet([]*types.BlockHeader{b})
if err != nil {
@ -465,14 +479,25 @@ func (cs *ChainStore) LoadTipSet(tsk types.TipSetKey) (*types.TipSet, error) {
return v.(*types.TipSet), nil
}
var blks []*types.BlockHeader
for _, c := range tsk.Cids() {
b, err := cs.GetBlock(c)
if err != nil {
return nil, xerrors.Errorf("get block %s: %w", c, err)
}
// Fetch tipset block headers from blockstore in parallel
var eg errgroup.Group
cids := tsk.Cids()
blks := make([]*types.BlockHeader, len(cids))
for i, c := range cids {
i, c := i, c
eg.Go(func() error {
b, err := cs.GetBlock(c)
if err != nil {
return xerrors.Errorf("get block %s: %w", c, err)
}
blks = append(blks, b)
blks[i] = b
return nil
})
}
err := eg.Wait()
if err != nil {
return nil, err
}
ts, err := types.NewTipSet(blks)
@ -1183,6 +1208,7 @@ func (cs *ChainStore) Export(ctx context.Context, ts *types.TipSet, inclRecentRo
}
blocksToWalk := ts.Cids()
currentMinHeight := ts.Height()
walkChain := func(blk cid.Cid) error {
if !seen.Visit(blk) {
@ -1203,6 +1229,13 @@ func (cs *ChainStore) Export(ctx context.Context, ts *types.TipSet, inclRecentRo
return xerrors.Errorf("unmarshaling block header (cid=%s): %w", blk, err)
}
if currentMinHeight > b.Height {
currentMinHeight = b.Height
if currentMinHeight%builtin.EpochsInDay == 0 {
log.Infow("export", "height", currentMinHeight)
}
}
var cids []cid.Cid
if !skipOldMsgs || b.Height > ts.Height()-inclRecentRoots {
mcids, err := recurseLinks(cs.bs, walked, b.Messages, []cid.Cid{b.Messages})
@ -1251,6 +1284,9 @@ func (cs *ChainStore) Export(ctx context.Context, ts *types.TipSet, inclRecentRo
return nil
}
log.Infow("export started")
exportStart := build.Clock.Now()
for len(blocksToWalk) > 0 {
next := blocksToWalk[0]
blocksToWalk = blocksToWalk[1:]
@ -1259,6 +1295,8 @@ func (cs *ChainStore) Export(ctx context.Context, ts *types.TipSet, inclRecentRo
}
}
log.Infow("export finished", "duration", build.Clock.Now().Sub(exportStart).Seconds())
return nil
}

73
chain/sync.go
View File

@ -597,7 +597,7 @@ func isPermanent(err error) bool {
return !errors.Is(err, ErrTemporal)
}
func (syncer *Syncer) ValidateTipSet(ctx context.Context, fts *store.FullTipSet) error {
func (syncer *Syncer) ValidateTipSet(ctx context.Context, fts *store.FullTipSet, useCache bool) error {
ctx, span := trace.StartSpan(ctx, "validateTipSet")
defer span.End()
@ -613,7 +613,7 @@ func (syncer *Syncer) ValidateTipSet(ctx context.Context, fts *store.FullTipSet)
b := b // rebind to a scoped variable
futures = append(futures, async.Err(func() error {
if err := syncer.ValidateBlock(ctx, b); err != nil {
if err := syncer.ValidateBlock(ctx, b, useCache); err != nil {
if isPermanent(err) {
syncer.bad.Add(b.Cid(), NewBadBlockReason([]cid.Cid{b.Cid()}, err.Error()))
}
@ -680,7 +680,7 @@ func blockSanityChecks(h *types.BlockHeader) error {
}
// ValidateBlock should match up with 'Semantical Validation' in validation.md in the spec
func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock) (err error) {
func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock, useCache bool) (err error) {
defer func() {
// b.Cid() could panic for empty blocks that are used in tests.
if rerr := recover(); rerr != nil {
@ -689,13 +689,15 @@ func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock) (er
}
}()
isValidated, err := syncer.store.IsBlockValidated(ctx, b.Cid())
if err != nil {
return xerrors.Errorf("check block validation cache %s: %w", b.Cid(), err)
}
if useCache {
isValidated, err := syncer.store.IsBlockValidated(ctx, b.Cid())
if err != nil {
return xerrors.Errorf("check block validation cache %s: %w", b.Cid(), err)
}
if isValidated {
return nil
if isValidated {
return nil
}
}
validationStart := build.Clock.Now()
@ -782,31 +784,35 @@ func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock) (er
b.Header.ParentWeight, pweight)
}
// Stuff that needs stateroot / worker address
stateroot, precp, err := syncer.sm.TipSetState(ctx, baseTs)
if err != nil {
return xerrors.Errorf("get tipsetstate(%d, %s) failed: %w", h.Height, h.Parents, err)
}
if stateroot != h.ParentStateRoot {
msgs, err := syncer.store.MessagesForTipset(baseTs)
stateRootCheck := async.Err(func() error {
stateroot, precp, err := syncer.sm.TipSetState(ctx, baseTs)
if err != nil {
log.Error("failed to load messages for tipset during tipset state mismatch error: ", err)
} else {
log.Warn("Messages for tipset with mismatching state:")
for i, m := range msgs {
mm := m.VMMessage()
log.Warnf("Message[%d]: from=%s to=%s method=%d params=%x", i, mm.From, mm.To, mm.Method, mm.Params)
}
return xerrors.Errorf("get tipsetstate(%d, %s) failed: %w", h.Height, h.Parents, err)
}
return xerrors.Errorf("parent state root did not match computed state (%s != %s)", stateroot, h.ParentStateRoot)
}
if stateroot != h.ParentStateRoot {
msgs, err := syncer.store.MessagesForTipset(baseTs)
if err != nil {
log.Error("failed to load messages for tipset during tipset state mismatch error: ", err)
} else {
log.Warn("Messages for tipset with mismatching state:")
for i, m := range msgs {
mm := m.VMMessage()
log.Warnf("Message[%d]: from=%s to=%s method=%d params=%x", i, mm.From, mm.To, mm.Method, mm.Params)
}
}
if precp != h.ParentMessageReceipts {
return xerrors.Errorf("parent receipts root did not match computed value (%s != %s)", precp, h.ParentMessageReceipts)
}
return xerrors.Errorf("parent state root did not match computed state (%s != %s)", stateroot, h.ParentStateRoot)
}
if precp != h.ParentMessageReceipts {
return xerrors.Errorf("parent receipts root did not match computed value (%s != %s)", precp, h.ParentMessageReceipts)
}
return nil
})
// Stuff that needs worker address
waddr, err := stmgr.GetMinerWorkerRaw(ctx, syncer.sm, lbst, h.Miner)
if err != nil {
return xerrors.Errorf("GetMinerWorkerRaw failed: %w", err)
@ -927,6 +933,7 @@ func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock) (er
winnerCheck,
msgsCheck,
baseFeeCheck,
stateRootCheck,
}
var merr error
@ -954,8 +961,10 @@ func (syncer *Syncer) ValidateBlock(ctx context.Context, b *types.FullBlock) (er
return mulErr
}
if err := syncer.store.MarkBlockAsValidated(ctx, b.Cid()); err != nil {
return xerrors.Errorf("caching block validation %s: %w", b.Cid(), err)
if useCache {
if err := syncer.store.MarkBlockAsValidated(ctx, b.Cid()); err != nil {
return xerrors.Errorf("caching block validation %s: %w", b.Cid(), err)
}
}
return nil
@ -1457,7 +1466,7 @@ func (syncer *Syncer) syncMessagesAndCheckState(ctx context.Context, headers []*
return syncer.iterFullTipsets(ctx, headers, func(ctx context.Context, fts *store.FullTipSet) error {
log.Debugw("validating tipset", "height", fts.TipSet().Height(), "size", len(fts.TipSet().Cids()))
if err := syncer.ValidateTipSet(ctx, fts); err != nil {
if err := syncer.ValidateTipSet(ctx, fts, true); err != nil {
log.Errorf("failed to validate tipset: %+v", err)
return xerrors.Errorf("message processing failed: %w", err)
}

4
chain/sync_test.go
View File

@ -732,7 +732,7 @@ func TestSyncInputs(t *testing.T) {
err := s.ValidateBlock(context.TODO(), &types.FullBlock{
Header: &types.BlockHeader{},
})
}, false)
if err == nil {
t.Fatal("should error on empty block")
}
@ -741,7 +741,7 @@ func TestSyncInputs(t *testing.T) {
h.ElectionProof = nil
err = s.ValidateBlock(context.TODO(), &types.FullBlock{Header: h})
err = s.ValidateBlock(context.TODO(), &types.FullBlock{Header: h}, false)
if err == nil {
t.Fatal("should error on block with nil election proof")
}

10
chain/types/tipset.go
View File

@ -167,12 +167,16 @@ func (ts *TipSet) Equals(ots *TipSet) bool {
return false
}
if len(ts.blks) != len(ots.blks) {
if ts.height != ots.height {
return false
}
for i, b := range ts.blks {
if b.Cid() != ots.blks[i].Cid() {
if len(ts.cids) != len(ots.cids) {
return false
}
for i, cid := range ts.cids {
if cid != ots.cids[i] {
return false
}
}

40
chain/types_test.go
View File

@ -1,9 +1,12 @@
package chain
import (
"crypto/rand"
"encoding/json"
"testing"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/lotus/chain/types"
)
@ -35,3 +38,40 @@ func TestSignedMessageJsonRoundtrip(t *testing.T) {
t.Fatal(err)
}
}
func TestAddressType(t *testing.T) {
build.SetAddressNetwork(address.Testnet)
addr, err := makeRandomAddress()
if err != nil {
t.Fatal(err)
}
if string(addr[0]) != address.TestnetPrefix {
t.Fatalf("address should start with %s", address.TestnetPrefix)
}
build.SetAddressNetwork(address.Mainnet)
addr, err = makeRandomAddress()
if err != nil {
t.Fatal(err)
}
if string(addr[0]) != address.MainnetPrefix {
t.Fatalf("address should start with %s", address.MainnetPrefix)
}
}
func makeRandomAddress() (string, error) {
bytes := make([]byte, 32)
_, err := rand.Read(bytes)
if err != nil {
return "", err
}
addr, err := address.NewActorAddress(bytes)
if err != nil {
return "", err
}
return addr.String(), nil
}

58
chain/vm/runtime.go
View File

@ -5,7 +5,6 @@ import (
"context"
"encoding/binary"
"fmt"
gruntime "runtime"
"time"
"github.com/filecoin-project/go-address"
@ -460,8 +459,10 @@ func (rt *Runtime) stateCommit(oldh, newh cid.Cid) aerrors.ActorError {
}
func (rt *Runtime) finilizeGasTracing() {
if rt.lastGasCharge != nil {
rt.lastGasCharge.TimeTaken = time.Since(rt.lastGasChargeTime)
if enableTracing {
if rt.lastGasCharge != nil {
rt.lastGasCharge.TimeTaken = time.Since(rt.lastGasChargeTime)
}
}
}
@ -490,34 +491,39 @@ func (rt *Runtime) chargeGasFunc(skip int) func(GasCharge) {
}
var enableTracing = false
func (rt *Runtime) chargeGasInternal(gas GasCharge, skip int) aerrors.ActorError {
toUse := gas.Total()
var callers [10]uintptr
cout := gruntime.Callers(2+skip, callers[:])
if enableTracing {
var callers [10]uintptr
now := build.Clock.Now()
if rt.lastGasCharge != nil {
rt.lastGasCharge.TimeTaken = now.Sub(rt.lastGasChargeTime)
cout := 0 //gruntime.Callers(2+skip, callers[:])
now := build.Clock.Now()
if rt.lastGasCharge != nil {
rt.lastGasCharge.TimeTaken = now.Sub(rt.lastGasChargeTime)
}
gasTrace := types.GasTrace{
Name: gas.Name,
Extra: gas.Extra,
TotalGas: toUse,
ComputeGas: gas.ComputeGas,
StorageGas: gas.StorageGas,
TotalVirtualGas: gas.VirtualCompute*GasComputeMulti + gas.VirtualStorage*GasStorageMulti,
VirtualComputeGas: gas.VirtualCompute,
VirtualStorageGas: gas.VirtualStorage,
Callers: callers[:cout],
}
rt.executionTrace.GasCharges = append(rt.executionTrace.GasCharges, &gasTrace)
rt.lastGasChargeTime = now
rt.lastGasCharge = &gasTrace
}
gasTrace := types.GasTrace{
Name: gas.Name,
Extra: gas.Extra,
TotalGas: toUse,
ComputeGas: gas.ComputeGas,
StorageGas: gas.StorageGas,
TotalVirtualGas: gas.VirtualCompute*GasComputeMulti + gas.VirtualStorage*GasStorageMulti,
VirtualComputeGas: gas.VirtualCompute,
VirtualStorageGas: gas.VirtualStorage,
Callers: callers[:cout],
}
rt.executionTrace.GasCharges = append(rt.executionTrace.GasCharges, &gasTrace)
rt.lastGasChargeTime = now
rt.lastGasCharge = &gasTrace
// overflow safe
if rt.gasUsed > rt.gasAvailable-toUse {
rt.gasUsed = rt.gasAvailable

17
chain/vm/vm.go
View File

@ -227,14 +227,21 @@ func (vm *VM) send(ctx context.Context, msg *types.Message, parent *Runtime,
}
rt := vm.makeRuntime(ctx, msg, origin, on, gasUsed, nac)
rt.lastGasChargeTime = start
if enableTracing {
rt.lastGasChargeTime = start
if parent != nil {
rt.lastGasChargeTime = parent.lastGasChargeTime
rt.lastGasCharge = parent.lastGasCharge
defer func() {
parent.lastGasChargeTime = rt.lastGasChargeTime
parent.lastGasCharge = rt.lastGasCharge
}()
}
}
if parent != nil {
rt.lastGasChargeTime = parent.lastGasChargeTime
rt.lastGasCharge = parent.lastGasCharge
defer func() {
parent.gasUsed = rt.gasUsed
parent.lastGasChargeTime = rt.lastGasChargeTime
parent.lastGasCharge = rt.lastGasCharge
}()
}
if gasCharge != nil {

37
cli/state.go
View File

@ -19,6 +19,7 @@ import (
"github.com/multiformats/go-multiaddr"
"github.com/ipfs/go-cid"
cbor "github.com/ipfs/go-ipld-cbor"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/multiformats/go-multihash"
"github.com/urfave/cli/v2"
@ -33,7 +34,9 @@ import (
"github.com/filecoin-project/specs-actors/actors/builtin/exported"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/api/apibstore"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain/state"
"github.com/filecoin-project/lotus/chain/stmgr"
"github.com/filecoin-project/lotus/chain/types"
)
@ -818,6 +821,10 @@ var stateComputeStateCmd = &cli.Command{
Name: "html",
Usage: "generate html report",
},
&cli.BoolFlag{
Name: "json",
Usage: "generate json output",
},
},
Action: func(cctx *cli.Context) error {
api, closer, err := GetFullNodeAPI(cctx)
@ -834,14 +841,14 @@ var stateComputeStateCmd = &cli.Command{
}
h := abi.ChainEpoch(cctx.Uint64("vm-height"))
if h == 0 {
if ts == nil {
head, err := api.ChainHead(ctx)
if err != nil {
return err
}
ts = head
if ts == nil {
head, err := api.ChainHead(ctx)
if err != nil {
return err
}
ts = head
}
if h == 0 {
h = ts.Height()
}
@ -862,14 +869,28 @@ var stateComputeStateCmd = &cli.Command{
return err
}
if cctx.Bool("json") {
out, err := json.Marshal(stout)
if err != nil {
return err
}
fmt.Println(string(out))
return nil
}
if cctx.Bool("html") {
st, err := state.LoadStateTree(cbor.NewCborStore(apibstore.NewAPIBlockstore(api)), stout.Root)
if err != nil {
return xerrors.Errorf("loading state tree: %w", err)
}
codeCache := map[address.Address]cid.Cid{}
getCode := func(addr address.Address) (cid.Cid, error) {
if c, found := codeCache[addr]; found {
return c, nil
}
c, err := api.StateGetActor(ctx, addr, ts.Key())
c, err := st.GetActor(addr)
if err != nil {
return cid.Cid{}, err
}

65
cli/wallet.go
View File

@ -9,13 +9,16 @@ import (
"os"
"strings"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/crypto"
types "github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/wallet"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
"github.com/urfave/cli/v2"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/big"
"github.com/filecoin-project/go-state-types/crypto"
types "github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/wallet"
"github.com/filecoin-project/lotus/lib/tablewriter"
)
var walletCmd = &cli.Command{
@ -66,6 +69,13 @@ var walletNew = &cli.Command{
var walletList = &cli.Command{
Name: "list",
Usage: "List wallet address",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "addr-only",
Usage: "Only print addresses",
Aliases: []string{"a"},
},
},
Action: func(cctx *cli.Context) error {
api, closer, err := GetFullNodeAPI(cctx)
if err != nil {
@ -79,9 +89,52 @@ var walletList = &cli.Command{
return err
}
// Assume an error means no default key is set
def, _ := api.WalletDefaultAddress(ctx)
tw := tablewriter.New(
tablewriter.Col("Address"),
tablewriter.Col("Balance"),
tablewriter.Col("Nonce"),
tablewriter.Col("Default"),
tablewriter.NewLineCol("Error"))
for _, addr := range addrs {
fmt.Println(addr.String())
if cctx.Bool("addr-only") {
fmt.Println(addr.String())
} else {
a, err := api.StateGetActor(ctx, addr, types.EmptyTSK)
if err != nil {
if !strings.Contains(err.Error(), "actor not found") {
tw.Write(map[string]interface{}{
"Address": addr,
"Error": err,
})
continue
}
a = &types.Actor{
Balance: big.Zero(),
}
}
row := map[string]interface{}{
"Address": addr,
"Balance": types.FIL(a.Balance),
"Nonce": a.Nonce,
}
if addr == def {
row["Default"] = "X"
}
tw.Write(row)
}
}
if !cctx.Bool("addr-only") {
return tw.Flush(os.Stdout)
}
return nil
},
}

98
cmd/lotus-bench/caching_verifier.go Normal file
View File

@ -0,0 +1,98 @@
package main
import (
"bufio"
"context"
"errors"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper"
"github.com/filecoin-project/specs-actors/actors/runtime/proof"
"github.com/ipfs/go-datastore"
"github.com/minio/blake2b-simd"
cbg "github.com/whyrusleeping/cbor-gen"
)
type cachingVerifier struct {
ds datastore.Datastore
backend ffiwrapper.Verifier
}
const bufsize = 128
func (cv cachingVerifier) withCache(execute func() (bool, error), param cbg.CBORMarshaler) (bool, error) {
hasher := blake2b.New256()
wr := bufio.NewWriterSize(hasher, bufsize)
err := param.MarshalCBOR(wr)
if err != nil {
log.Errorf("could not marshal call info: %+v", err)
return execute()
}
err = wr.Flush()
if err != nil {
log.Errorf("could not flush: %+v", err)
return execute()
}
hash := hasher.Sum(nil)
key := datastore.NewKey(string(hash))
fromDs, err := cv.ds.Get(key)
if err == nil {
switch fromDs[0] {
case 's':
return true, nil
case 'f':
return false, nil
case 'e':
return false, errors.New(string(fromDs[1:]))
default:
log.Errorf("bad cached result in cache %s(%x)", fromDs[0], fromDs[0])
return execute()
}
} else if errors.Is(err, datastore.ErrNotFound) {
// recalc
ok, err := execute()
var save []byte
if err != nil {
if ok {
log.Errorf("success with an error: %+v", err)
} else {
save = append([]byte{'e'}, []byte(err.Error())...)
}
} else if ok {
save = []byte{'s'}
} else {
save = []byte{'f'}
}
if len(save) != 0 {
errSave := cv.ds.Put(key, save)
if errSave != nil {
log.Errorf("error saving result: %+v", errSave)
}
}
return ok, err
} else {
log.Errorf("could not get data from cache: %+v", err)
return execute()
}
}
func (cv *cachingVerifier) VerifySeal(svi proof.SealVerifyInfo) (bool, error) {
return cv.withCache(func() (bool, error) {
return cv.backend.VerifySeal(svi)
}, &svi)
}
func (cv *cachingVerifier) VerifyWinningPoSt(ctx context.Context, info proof.WinningPoStVerifyInfo) (bool, error) {
return cv.backend.VerifyWinningPoSt(ctx, info)
}
func (cv *cachingVerifier) VerifyWindowPoSt(ctx context.Context, info proof.WindowPoStVerifyInfo) (bool, error) {
return cv.withCache(func() (bool, error) {
return cv.backend.VerifyWindowPoSt(ctx, info)
}, &info)
}
func (cv *cachingVerifier) GenerateWinningPoStSectorChallenge(ctx context.Context, proofType abi.RegisteredPoStProof, a abi.ActorID, rnd abi.PoStRandomness, u uint64) ([]uint64, error) {
return cv.backend.GenerateWinningPoStSectorChallenge(ctx, proofType, a, rnd, u)
}
var _ ffiwrapper.Verifier = (*cachingVerifier)(nil)

208
cmd/lotus-bench/import.go
View File

@ -16,6 +16,8 @@ import (
"sort"
"time"
"github.com/cockroachdb/pebble"
"github.com/cockroachdb/pebble/bloom"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/chain/stmgr"
"github.com/filecoin-project/lotus/chain/store"
@ -24,12 +26,16 @@ import (
"github.com/filecoin-project/lotus/lib/blockstore"
_ "github.com/filecoin-project/lotus/lib/sigs/bls"
_ "github.com/filecoin-project/lotus/lib/sigs/secp"
"github.com/ipld/go-car"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper"
bdg "github.com/dgraph-io/badger/v2"
"github.com/ipfs/go-datastore"
badger "github.com/ipfs/go-ds-badger2"
pebbleds "github.com/ipfs/go-ds-pebble"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
)
@ -60,6 +66,29 @@ var importBenchCmd = &cli.Command{
Name: "repodir",
Usage: "set the repo directory for the lotus bench run (defaults to /tmp)",
},
&cli.StringFlag{
Name: "syscall-cache",
Usage: "read and write syscall results from datastore",
},
&cli.BoolFlag{
Name: "export-traces",
Usage: "should we export execution traces",
Value: true,
},
&cli.BoolFlag{
Name: "no-import",
Usage: "should we import the chain? if set to true chain has to be previously imported",
},
&cli.BoolFlag{
Name: "global-profile",
Value: true,
},
&cli.Int64Flag{
Name: "start-at",
},
&cli.BoolFlag{
Name: "only-import",
},
},
Action: func(cctx *cli.Context) error {
vm.BatchSealVerifyParallelism = cctx.Int("batch-seal-verify-threads")
@ -74,6 +103,10 @@ var importBenchCmd = &cli.Command{
}
defer cfi.Close() //nolint:errcheck // read only file
go func() {
http.ListenAndServe("localhost:6060", nil) //nolint:errcheck
}()
var tdir string
if rdir := cctx.String("repodir"); rdir != "" {
tdir = rdir
@ -85,33 +118,105 @@ var importBenchCmd = &cli.Command{
tdir = tmp
}
bds, err := badger.NewDatastore(tdir, nil)
bdgOpt := badger.DefaultOptions
bdgOpt.GcInterval = 0
bdgOpt.Options = bdg.DefaultOptions("")
bdgOpt.Options.SyncWrites = false
bdgOpt.Options.Truncate = true
bdgOpt.Options.DetectConflicts = false
var bds datastore.Batching
if false {
cache := 512
bds, err = pebbleds.NewDatastore(tdir, &pebble.Options{
// Pebble has a single combined cache area and the write
// buffers are taken from this too. Assign all available
// memory allowance for cache.
Cache: pebble.NewCache(int64(cache * 1024 * 1024)),
// The size of memory table(as well as the write buffer).
// Note, there may have more than two memory tables in the system.
// MemTableStopWritesThreshold can be configured to avoid the memory abuse.
MemTableSize: cache * 1024 * 1024 / 4,
// The default compaction concurrency(1 thread),
// Here use all available CPUs for faster compaction.
MaxConcurrentCompactions: runtime.NumCPU(),
// Per-level options. Options for at least one level must be specified. The
// options for the last level are used for all subsequent levels.
Levels: []pebble.LevelOptions{
{TargetFileSize: 16 * 1024 * 1024, FilterPolicy: bloom.FilterPolicy(10), Compression: pebble.NoCompression},
},
Logger: log,
})
} else {
bds, err = badger.NewDatastore(tdir, &bdgOpt)
}
if err != nil {
return err
}
defer bds.Close() //nolint:errcheck
bs := blockstore.NewBlockstore(bds)
cbs, err := blockstore.CachedBlockstore(context.TODO(), bs, blockstore.DefaultCacheOpts())
cacheOpts := blockstore.DefaultCacheOpts()
cacheOpts.HasBloomFilterSize = 0
cbs, err := blockstore.CachedBlockstore(context.TODO(), bs, cacheOpts)
if err != nil {
return err
}
bs = cbs
ds := datastore.NewMapDatastore()
cs := store.NewChainStore(bs, ds, vm.Syscalls(ffiwrapper.ProofVerifier))
var verifier ffiwrapper.Verifier = ffiwrapper.ProofVerifier
if cctx.IsSet("syscall-cache") {
scds, err := badger.NewDatastore(cctx.String("syscall-cache"), &bdgOpt)
if err != nil {
return xerrors.Errorf("opening syscall-cache datastore: %w", err)
}
defer scds.Close() //nolint:errcheck
verifier = &cachingVerifier{
ds: scds,
backend: verifier,
}
}
if cctx.Bool("only-gc") {
return nil
}
cs := store.NewChainStore(bs, ds, vm.Syscalls(verifier))
stm := stmgr.NewStateManager(cs)
prof, err := os.Create("import-bench.prof")
if err != nil {
return err
}
defer prof.Close() //nolint:errcheck
if cctx.Bool("global-profile") {
prof, err := os.Create("import-bench.prof")
if err != nil {
return err
}
defer prof.Close() //nolint:errcheck
if err := pprof.StartCPUProfile(prof); err != nil {
return err
if err := pprof.StartCPUProfile(prof); err != nil {
return err
}
}
head, err := cs.Import(cfi)
if err != nil {
return err
var head *types.TipSet
if !cctx.Bool("no-import") {
head, err = cs.Import(cfi)
if err != nil {
return err
}
} else {
cr, err := car.NewCarReader(cfi)
if err != nil {
return err
}
head, err = cs.LoadTipSet(types.NewTipSetKey(cr.Header.Roots...))
if err != nil {
return err
}
}
if cctx.Bool("only-import") {
return nil
}
gb, err := cs.GetTipsetByHeight(context.TODO(), 0, head, true)
@ -124,6 +229,20 @@ var importBenchCmd = &cli.Command{
return err
}
startEpoch := abi.ChainEpoch(1)
if cctx.IsSet("start-at") {
startEpoch = abi.ChainEpoch(cctx.Int64("start-at"))
start, err := cs.GetTipsetByHeight(context.TODO(), abi.ChainEpoch(cctx.Int64("start-at")), head, true)
if err != nil {
return err
}
err = cs.SetHead(start)
if err != nil {
return err
}
}
if h := cctx.Int64("height"); h != 0 {
tsh, err := cs.GetTipsetByHeight(context.TODO(), abi.ChainEpoch(h), head, true)
if err != nil {
@ -134,7 +253,7 @@ var importBenchCmd = &cli.Command{
ts := head
tschain := []*types.TipSet{ts}
for ts.Height() != 0 {
for ts.Height() > startEpoch {
next, err := cs.LoadTipSet(ts.Parents())
if err != nil {
return err
@ -144,45 +263,48 @@ var importBenchCmd = &cli.Command{
ts = next
}
ibj, err := os.Create("import-bench.json")
if err != nil {
return err
var enc *json.Encoder
if cctx.Bool("export-traces") {
ibj, err := os.Create("import-bench.json")
if err != nil {
return err
}
defer ibj.Close() //nolint:errcheck
enc = json.NewEncoder(ibj)
}
defer ibj.Close() //nolint:errcheck
enc := json.NewEncoder(ibj)
var lastTse *TipSetExec
lastState := tschain[len(tschain)-1].ParentState()
for i := len(tschain) - 2; i >= 0; i-- {
for i := len(tschain) - 1; i >= 1; i-- {
cur := tschain[i]
start := time.Now()
log.Infof("computing state (height: %d, ts=%s)", cur.Height(), cur.Cids())
if cur.ParentState() != lastState {
lastTrace := lastTse.Trace
st, trace, err := stm.ExecutionTrace(context.TODO(), cur)
if err != nil {
return err
}
tse := &TipSetExec{
TipSet: cur.Key(),
Trace: trace,
Duration: time.Since(start),
}
if enc != nil {
stripCallers(tse.Trace)
if err := enc.Encode(tse); err != nil {
return xerrors.Errorf("failed to write out tipsetexec: %w", err)
}
}
if tschain[i-1].ParentState() != st {
stripCallers(tse.Trace)
lastTrace := tse.Trace
d, err := json.MarshalIndent(lastTrace, "", " ")
if err != nil {
panic(err)
}
fmt.Println("TRACE")
fmt.Println(string(d))
return xerrors.Errorf("tipset chain had state mismatch at height %d (%s != %s)", cur.Height(), cur.ParentState(), lastState)
}
start := time.Now()
st, trace, err := stm.ExecutionTrace(context.TODO(), cur)
if err != nil {
return err
}
stripCallers(trace)
lastTse = &TipSetExec{
TipSet: cur.Key(),
Trace: trace,
Duration: time.Since(start),
}
lastState = st
if err := enc.Encode(lastTse); err != nil {
return xerrors.Errorf("failed to write out tipsetexec: %w", err)
//fmt.Println(statediff.Diff(context.Background(), bs, tschain[i-1].ParentState(), st, statediff.ExpandActors))
return xerrors.Errorf("tipset chain had state mismatch at height %d (%s != %s)", cur.Height(), cur.ParentState(), st)
}
}

4
cmd/lotus-bench/main.go
View File

@ -76,8 +76,6 @@ func main() {
log.Info("Starting lotus-bench")
policy.AddSupportedProofTypes(abi.RegisteredSealProof_StackedDrg2KiBV1)
app := &cli.App{
Name: "lotus-bench",
Usage: "Benchmark performance of lotus on your hardware",
@ -147,6 +145,8 @@ var sealBenchCmd = &cli.Command{
},
},
Action: func(c *cli.Context) error {
policy.AddSupportedProofTypes(abi.RegisteredSealProof_StackedDrg2KiBV1)
if c.Bool("no-gpu") {
err := os.Setenv("BELLMAN_NO_GPU", "1")
if err != nil {

7
cmd/lotus-seal-worker/main.go
View File

@ -109,6 +109,11 @@ var runCmd = &cli.Command{
Name: "no-local-storage",
Usage: "don't use storageminer repo for sector storage",
},
&cli.BoolFlag{
Name: "no-swap",
Usage: "don't use swap",
Value: false,
},
&cli.BoolFlag{
Name: "addpiece",
Usage: "enable addpiece",
@ -346,6 +351,7 @@ var runCmd = &cli.Command{
LocalWorker: sectorstorage.NewLocalWorker(sectorstorage.WorkerConfig{
SealProof: spt,
TaskTypes: taskTypes,
NoSwap: cctx.Bool("no-swap"),
}, remote, localStore, nodeApi),
localStore: localStore,
ls: lr,
@ -465,6 +471,7 @@ func watchMinerConn(ctx context.Context, cctx *cli.Context, nodeApi api.StorageM
"run",
fmt.Sprintf("--listen=%s", cctx.String("listen")),
fmt.Sprintf("--no-local-storage=%t", cctx.Bool("no-local-storage")),
fmt.Sprintf("--no-swap=%t", cctx.Bool("no-swap")),
fmt.Sprintf("--addpiece=%t", cctx.Bool("addpiece")),
fmt.Sprintf("--precommit1=%t", cctx.Bool("precommit1")),
fmt.Sprintf("--unseal=%t", cctx.Bool("unseal")),

138
cmd/lotus-shed/balances.go
View File

@ -3,9 +3,16 @@ package main
import (
"context"
"fmt"
"strconv"
"github.com/docker/go-units"
lotusbuiltin "github.com/filecoin-project/lotus/chain/actors/builtin"
"github.com/filecoin-project/lotus/chain/actors/builtin/power"
"github.com/filecoin-project/lotus/chain/actors/builtin/reward"
"github.com/ipfs/go-cid"
cbor "github.com/ipfs/go-ipld-cbor"
logging "github.com/ipfs/go-log/v2"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
@ -44,6 +51,7 @@ var auditsCmd = &cli.Command{
Subcommands: []*cli.Command{
chainBalanceCmd,
chainBalanceStateCmd,
chainPledgeCmd,
},
}
@ -248,3 +256,133 @@ var chainBalanceStateCmd = &cli.Command{
return nil
},
}
var chainPledgeCmd = &cli.Command{
Name: "stateroot-pledge",
Description: "Calculate sector pledge numbers",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "repo",
Value: "~/.lotus",
},
},
ArgsUsage: "[stateroot epoch]",
Action: func(cctx *cli.Context) error {
logging.SetLogLevel("badger", "ERROR")
ctx := context.TODO()
if !cctx.Args().Present() {
return fmt.Errorf("must pass state root")
}
sroot, err := cid.Decode(cctx.Args().First())
if err != nil {
return fmt.Errorf("failed to parse input: %w", err)
}
epoch, err := strconv.ParseInt(cctx.Args().Get(1), 10, 64)
if err != nil {
return xerrors.Errorf("parsing epoch arg: %w", err)
}
fsrepo, err := repo.NewFS(cctx.String("repo"))
if err != nil {
return err
}
lkrepo, err := fsrepo.Lock(repo.FullNode)
if err != nil {
return err
}
defer lkrepo.Close() //nolint:errcheck
ds, err := lkrepo.Datastore("/chain")
if err != nil {
return err
}
mds, err := lkrepo.Datastore("/metadata")
if err != nil {
return err
}
bs := blockstore.NewBlockstore(ds)
cs := store.NewChainStore(bs, mds, vm.Syscalls(ffiwrapper.ProofVerifier))
cst := cbor.NewCborStore(bs)
store := adt.WrapStore(ctx, cst)
sm := stmgr.NewStateManager(cs)
state, err := state.LoadStateTree(cst, sroot)
if err != nil {
return err
}
var (
powerSmoothed lotusbuiltin.FilterEstimate
pledgeCollateral abi.TokenAmount
)
if act, err := state.GetActor(power.Address); err != nil {
return xerrors.Errorf("loading miner actor: %w", err)
} else if s, err := power.Load(store, act); err != nil {
return xerrors.Errorf("loading power actor state: %w", err)
} else if p, err := s.TotalPowerSmoothed(); err != nil {
return xerrors.Errorf("failed to determine total power: %w", err)
} else if c, err := s.TotalLocked(); err != nil {
return xerrors.Errorf("failed to determine pledge collateral: %w", err)
} else {
powerSmoothed = p
pledgeCollateral = c
}
circ, err := sm.GetCirculatingSupplyDetailed(ctx, abi.ChainEpoch(epoch), state)
if err != nil {
return err
}
fmt.Println("(real) circulating supply: ", types.FIL(circ.FilCirculating))
if circ.FilCirculating.LessThan(big.Zero()) {
circ.FilCirculating = big.Zero()
}
rewardActor, err := state.GetActor(reward.Address)
if err != nil {
return xerrors.Errorf("loading miner actor: %w", err)
}
rewardState, err := reward.Load(store, rewardActor)
if err != nil {
return xerrors.Errorf("loading reward actor state: %w", err)
}
fmt.Println("FilVested", types.FIL(circ.FilVested))
fmt.Println("FilMined", types.FIL(circ.FilMined))
fmt.Println("FilBurnt", types.FIL(circ.FilBurnt))
fmt.Println("FilLocked", types.FIL(circ.FilLocked))
fmt.Println("FilCirculating", types.FIL(circ.FilCirculating))
for _, sectorWeight := range []abi.StoragePower{
types.NewInt(32 << 30),
types.NewInt(64 << 30),
types.NewInt(32 << 30 * 10),
types.NewInt(64 << 30 * 10),
} {
initialPledge, err := rewardState.InitialPledgeForPower(
sectorWeight,
pledgeCollateral,
&powerSmoothed,
circ.FilCirculating,
)
if err != nil {
return xerrors.Errorf("calculating initial pledge: %w", err)
}
fmt.Println("IP ", units.HumanSize(float64(sectorWeight.Uint64())), types.FIL(initialPledge))
}
return nil
},
}

268
cmd/lotus-shed/dealtracker.go
View File

@ -5,10 +5,10 @@ import (
"encoding/json"
"net"
"net/http"
"os"
"strings"
"sync"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/api"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/ipfs/go-cid"
@ -19,58 +19,54 @@ type dealStatsServer struct {
api api.FullNode
}
var filteredClients map[address.Address]bool
// Requested by @jbenet
// How many epochs back to look at for dealstats
var epochLookback = abi.ChainEpoch(10)
// these lists grow continuously with the network
// TODO: need to switch this to an LRU of sorts, to ensure refreshes
var knownFiltered = new(sync.Map)
var resolvedWallets = new(sync.Map)
func init() {
fc := []string{"t0112", "t0113", "t0114", "t010089"}
filtered, set := os.LookupEnv("FILTERED_CLIENTS")
if set {
fc = strings.Split(filtered, ":")
}
filteredClients = make(map[address.Address]bool)
for _, a := range fc {
addr, err := address.NewFromString(a)
for _, a := range []string{
"t0100", // client for genesis miner
"t0101", // client for genesis miner
"t0102", // client for genesis miner
"t0112", // client for genesis miner
"t0113", // client for genesis miner
"t0114", // client for genesis miner
"t1nslxql4pck5pq7hddlzym3orxlx35wkepzjkm3i", // SR1 dealbot wallet
"t1stghxhdp2w53dym2nz2jtbpk6ccd4l2lxgmezlq", // SR1 dealbot wallet
"t1mcr5xkgv4jdl3rnz77outn6xbmygb55vdejgbfi", // SR1 dealbot wallet
"t1qiqdbbmrdalbntnuapriirduvxu5ltsc5mhy7si", // SR1 dealbot wallet
} {
a, err := address.NewFromString(a)
if err != nil {
panic(err)
}
filteredClients[addr] = true
knownFiltered.Store(a, true)
}
}
type dealCountResp struct {
Total int64 `json:"total"`
Epoch int64 `json:"epoch"`
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload int64 `json:"payload"`
}
func (dss *dealStatsServer) handleStorageDealCount(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
head, err := dss.api.ChainHead(ctx)
if err != nil {
log.Warnf("failed to get chain head: %s", err)
epoch, deals := dss.filteredDealList()
if epoch == 0 {
w.WriteHeader(500)
return
}
deals, err := dss.api.StateMarketDeals(ctx, head.Key())
if err != nil {
log.Warnf("failed to get market deals: %s", err)
w.WriteHeader(500)
return
}
var count int64
for _, d := range deals {
if !filteredClients[d.Proposal.Client] {
count++
}
}
if err := json.NewEncoder(w).Encode(&dealCountResp{
Total: count,
Epoch: int64(head.Height()),
Endpoint: "COUNT_DEALS",
Payload: int64(len(deals)),
Epoch: epoch,
}); err != nil {
log.Warnf("failed to write back deal count response: %s", err)
return
@ -78,39 +74,28 @@ func (dss *dealStatsServer) handleStorageDealCount(w http.ResponseWriter, r *htt
}
type dealAverageResp struct {
AverageSize int64 `json:"average_size"`
Epoch int64 `json:"epoch"`
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload int64 `json:"payload"`
}
func (dss *dealStatsServer) handleStorageDealAverageSize(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
head, err := dss.api.ChainHead(ctx)
if err != nil {
log.Warnf("failed to get chain head: %s", err)
epoch, deals := dss.filteredDealList()
if epoch == 0 {
w.WriteHeader(500)
return
}
deals, err := dss.api.StateMarketDeals(ctx, head.Key())
if err != nil {
log.Warnf("failed to get market deals: %s", err)
w.WriteHeader(500)
return
}
var count int64
var totalBytes int64
for _, d := range deals {
if !filteredClients[d.Proposal.Client] {
count++
totalBytes += int64(d.Proposal.PieceSize.Unpadded())
}
totalBytes += int64(d.deal.Proposal.PieceSize.Unpadded())
}
if err := json.NewEncoder(w).Encode(&dealAverageResp{
AverageSize: totalBytes / count,
Epoch: int64(head.Height()),
Endpoint: "AVERAGE_DEAL_SIZE",
Payload: totalBytes / int64(len(deals)),
Epoch: epoch,
}); err != nil {
log.Warnf("failed to write back deal average response: %s", err)
return
@ -118,37 +103,27 @@ func (dss *dealStatsServer) handleStorageDealAverageSize(w http.ResponseWriter,
}
type dealTotalResp struct {
TotalBytes int64 `json:"total_size"`
Epoch int64 `json:"epoch"`
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload int64 `json:"payload"`
}
func (dss *dealStatsServer) handleStorageDealTotalReal(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
head, err := dss.api.ChainHead(ctx)
if err != nil {
log.Warnf("failed to get chain head: %s", err)
w.WriteHeader(500)
return
}
deals, err := dss.api.StateMarketDeals(ctx, head.Key())
if err != nil {
log.Warnf("failed to get market deals: %s", err)
epoch, deals := dss.filteredDealList()
if epoch == 0 {
w.WriteHeader(500)
return
}
var totalBytes int64
for _, d := range deals {
if !filteredClients[d.Proposal.Client] {
totalBytes += int64(d.Proposal.PieceSize.Unpadded())
}
totalBytes += int64(d.deal.Proposal.PieceSize.Unpadded())
}
if err := json.NewEncoder(w).Encode(&dealTotalResp{
TotalBytes: totalBytes,
Epoch: int64(head.Height()),
Endpoint: "DEAL_BYTES",
Payload: totalBytes,
Epoch: epoch,
}); err != nil {
log.Warnf("failed to write back deal average response: %s", err)
return
@ -157,6 +132,12 @@ func (dss *dealStatsServer) handleStorageDealTotalReal(w http.ResponseWriter, r
}
type clientStatsOutput struct {
Epoch int64 `json:"epoch"`
Endpoint string `json:"endpoint"`
Payload []*clientStats `json:"payload"`
}
type clientStats struct {
Client address.Address `json:"client"`
DataSize int64 `json:"data_size"`
NumCids int `json:"num_cids"`
@ -168,51 +149,41 @@ type clientStatsOutput struct {
}
func (dss *dealStatsServer) handleStorageClientStats(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
head, err := dss.api.ChainHead(ctx)
if err != nil {
log.Warnf("failed to get chain head: %s", err)
epoch, deals := dss.filteredDealList()
if epoch == 0 {
w.WriteHeader(500)
return
}
deals, err := dss.api.StateMarketDeals(ctx, head.Key())
if err != nil {
log.Warnf("failed to get market deals: %s", err)
w.WriteHeader(500)
return
}
stats := make(map[address.Address]*clientStatsOutput)
stats := make(map[address.Address]*clientStats)
for _, d := range deals {
if filteredClients[d.Proposal.Client] {
continue
}
st, ok := stats[d.Proposal.Client]
st, ok := stats[d.deal.Proposal.Client]
if !ok {
st = &clientStatsOutput{
Client: d.Proposal.Client,
st = &clientStats{
Client: d.resolvedWallet,
cids: make(map[cid.Cid]bool),
providers: make(map[address.Address]bool),
}
stats[d.Proposal.Client] = st
stats[d.deal.Proposal.Client] = st
}
st.DataSize += int64(d.Proposal.PieceSize.Unpadded())
st.cids[d.Proposal.PieceCID] = true
st.providers[d.Proposal.Provider] = true
st.DataSize += int64(d.deal.Proposal.PieceSize.Unpadded())
st.cids[d.deal.Proposal.PieceCID] = true
st.providers[d.deal.Proposal.Provider] = true
st.NumDeals++
}
out := make([]*clientStatsOutput, 0, len(stats))
for _, cso := range stats {
cso.NumCids = len(cso.cids)
cso.NumMiners = len(cso.providers)
out = append(out, cso)
out := clientStatsOutput{
Epoch: epoch,
Endpoint: "CLIENT_DEAL_STATS",
Payload: make([]*clientStats, 0, len(stats)),
}
for _, cs := range stats {
cs.NumCids = len(cs.cids)
cs.NumMiners = len(cs.providers)
out.Payload = append(out.Payload, cs)
}
if err := json.NewEncoder(w).Encode(out); err != nil {
@ -221,6 +192,93 @@ func (dss *dealStatsServer) handleStorageClientStats(w http.ResponseWriter, r *h
}
}
type dealInfo struct {
deal api.MarketDeal
resolvedWallet address.Address
}
// filteredDealList returns the current epoch and a list of filtered deals
// on error returns an epoch of 0
func (dss *dealStatsServer) filteredDealList() (int64, map[string]dealInfo) {
ctx := context.Background()
head, err := dss.api.ChainHead(ctx)
if err != nil {
log.Warnf("failed to get chain head: %s", err)
return 0, nil
}
head, err = dss.api.ChainGetTipSetByHeight(ctx, head.Height()-epochLookback, head.Key())
if err != nil {
log.Warnf("failed to walk back %s epochs: %s", epochLookback, err)
return 0, nil
}
// Disabled as per @pooja's request
//
// // Exclude any address associated with a miner
// miners, err := dss.api.StateListMiners(ctx, head.Key())
// if err != nil {
// log.Warnf("failed to get miner list: %s", err)
// return 0, nil
// }
// for _, m := range miners {
// info, err := dss.api.StateMinerInfo(ctx, m, head.Key())
// if err != nil {
// log.Warnf("failed to get info for known miner '%s': %s", m, err)
// continue
// }
// knownFiltered.Store(info.Owner, true)
// knownFiltered.Store(info.Worker, true)
// for _, a := range info.ControlAddresses {
// knownFiltered.Store(a, true)
// }
// }
deals, err := dss.api.StateMarketDeals(ctx, head.Key())
if err != nil {
log.Warnf("failed to get market deals: %s", err)
return 0, nil
}
ret := make(map[string]dealInfo, len(deals))
for dealKey, d := range deals {
// Counting no-longer-active deals as per Pooja's request
// // https://github.com/filecoin-project/specs-actors/blob/v0.9.9/actors/builtin/market/deal.go#L81-L85
// if d.State.SectorStartEpoch < 0 {
// continue
// }
if _, isFiltered := knownFiltered.Load(d.Proposal.Client); isFiltered {
continue
}
if _, wasSeen := resolvedWallets.Load(d.Proposal.Client); !wasSeen {
w, err := dss.api.StateAccountKey(ctx, d.Proposal.Client, head.Key())
if err != nil {
log.Warnf("failed to resolve id '%s' to wallet address: %s", d.Proposal.Client, err)
continue
} else {
resolvedWallets.Store(d.Proposal.Client, w)
}
}
w, _ := resolvedWallets.Load(d.Proposal.Client)
if _, isFiltered := knownFiltered.Load(w); isFiltered {
continue
}
ret[dealKey] = dealInfo{
deal: d,
resolvedWallet: w.(address.Address),
}
}
return int64(head.Height()), ret
}
var serveDealStatsCmd = &cli.Command{
Name: "serve-deal-stats",
Flags: []cli.Flag{},
@ -260,6 +318,8 @@ var serveDealStatsCmd = &cli.Command{
panic(err)
}
log.Warnf("deal-stat server listening on %s\n== NOTE: QUERIES ARE EXPENSIVE - YOU MUST FRONT-CACHE THIS SERVICE\n", list.Addr().String())
return s.Serve(list)
},
}

57
cmd/lotus-shed/import-car.go
View File

@ -1,10 +1,13 @@
package main
import (
"encoding/hex"
"fmt"
"io"
"os"
block "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
"github.com/ipld/go-car"
"github.com/urfave/cli/v2"
"golang.org/x/xerrors"
@ -81,3 +84,57 @@ var importCarCmd = &cli.Command{
}
},
}
var importObjectCmd = &cli.Command{
Name: "import-obj",
Usage: "import a raw ipld object into your datastore",
Action: func(cctx *cli.Context) error {
r, err := repo.NewFS(cctx.String("repo"))
if err != nil {
return xerrors.Errorf("opening fs repo: %w", err)
}
exists, err := r.Exists()
if err != nil {
return err
}
if !exists {
return xerrors.Errorf("lotus repo doesn't exist")
}
lr, err := r.Lock(repo.FullNode)
if err != nil {
return err
}
defer lr.Close() //nolint:errcheck
ds, err := lr.Datastore("/chain")
if err != nil {
return err
}
bs := blockstore.NewBlockstore(ds)
c, err := cid.Decode(cctx.Args().Get(0))
if err != nil {
return err
}
data, err := hex.DecodeString(cctx.Args().Get(1))
if err != nil {
return err
}
blk, err := block.NewBlockWithCid(data, c)
if err != nil {
return err
}
if err := bs.Put(blk); err != nil {
return err
}
return nil
},
}

2
cmd/lotus-shed/main.go
View File

@ -25,6 +25,7 @@ func main() {
staterootCmd,
auditsCmd,
importCarCmd,
importObjectCmd,
commpToCidCmd,
fetchParamCmd,
proofsCmd,
@ -37,6 +38,7 @@ func main() {
exportChainCmd,
consensusCmd,
serveDealStatsCmd,
syncCmd,
}
app := &cli.App{

64
cmd/lotus-shed/sync.go Normal file
View File

@ -0,0 +1,64 @@
package main
import (
"fmt"
"github.com/ipfs/go-cid"
"github.com/filecoin-project/lotus/chain/types"
lcli "github.com/filecoin-project/lotus/cli"
"github.com/urfave/cli/v2"
)
var syncCmd = &cli.Command{
Name: "sync",
Usage: "tools for diagnosing sync issues",
Flags: []cli.Flag{},
Subcommands: []*cli.Command{
syncValidateCmd,
},
}
var syncValidateCmd = &cli.Command{
Name: "validate",
Usage: "checks whether a provided tipset is valid",
Action: func(cctx *cli.Context) error {
api, closer, err := lcli.GetFullNodeAPI(cctx)
if err != nil {
return err
}
defer closer()
ctx := lcli.ReqContext(cctx)
if cctx.Args().Len() < 1 {
fmt.Println("usage: <blockCid1> <blockCid2>...")
fmt.Println("At least one block cid must be provided")
return nil
}
args := cctx.Args().Slice()
var tscids []cid.Cid
for _, s := range args {
c, err := cid.Decode(s)
if err != nil {
return fmt.Errorf("block cid was invalid: %s", err)
}
tscids = append(tscids, c)
}
tsk := types.NewTipSetKey(tscids...)
valid, err := api.SyncValidateTipset(ctx, tsk)
if err != nil {
fmt.Println("Tipset is invalid: ", err)
}
if valid {
fmt.Println("Tipset is valid")
}
return nil
},
}

16
cmd/lotus-storage-miner/info.go
View File

@ -33,6 +33,12 @@ var infoCmd = &cli.Command{
Subcommands: []*cli.Command{
infoAllCmd,
},
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "hide-sectors-info",
Usage: "hide sectors info",
},
},
Action: infoCmdAct,
}
@ -199,10 +205,12 @@ func infoCmdAct(cctx *cli.Context) error {
fmt.Printf("Expected Seal Duration: %s\n\n", sealdur)
fmt.Println("Sectors:")
err = sectorsInfo(ctx, nodeApi)
if err != nil {
return err
if !cctx.Bool("hide-sectors-info") {
fmt.Println("Sectors:")
err = sectorsInfo(ctx, nodeApi)
if err != nil {
return err
}
}
// TODO: grab actr state / info

78
cmd/lotus/daemon.go
View File

@ -3,11 +3,14 @@
package main
import (
"bufio"
"context"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"runtime/pprof"
"strings"
@ -23,6 +26,7 @@ import (
"go.opencensus.io/stats/view"
"go.opencensus.io/tag"
"golang.org/x/xerrors"
"gopkg.in/cheggaaa/pb.v1"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/build"
@ -100,11 +104,11 @@ var DaemonCmd = &cli.Command{
},
&cli.StringFlag{
Name: "import-chain",
Usage: "on first run, load chain from given file and validate",
Usage: "on first run, load chain from given file or url and validate",
},
&cli.StringFlag{
Name: "import-snapshot",
Usage: "import chain state from a given chain export file",
Usage: "import chain state from a given chain export file or url",
},
&cli.BoolFlag{
Name: "halt-after-import",
@ -123,6 +127,10 @@ var DaemonCmd = &cli.Command{
Usage: "manage open file limit",
Value: true,
},
&cli.StringFlag{
Name: "config",
Usage: "specify path of config file to use",
},
},
Action: func(cctx *cli.Context) error {
err := runmetrics.Enable(runmetrics.RunMetricOptions{
@ -176,6 +184,10 @@ var DaemonCmd = &cli.Command{
return xerrors.Errorf("opening fs repo: %w", err)
}
if cctx.String("config") != "" {
r.SetConfigPath(cctx.String("config"))
}
if err := r.Init(repo.FullNode); err != nil && err != repo.ErrRepoExists {
return xerrors.Errorf("repo init error: %w", err)
}
@ -206,11 +218,6 @@ var DaemonCmd = &cli.Command{
issnapshot = true
}
chainfile, err := homedir.Expand(chainfile)
if err != nil {
return err
}
if err := ImportChain(r, chainfile, issnapshot); err != nil {
return err
}
@ -326,12 +333,42 @@ func importKey(ctx context.Context, api api.FullNode, f string) error {
return nil
}
func ImportChain(r repo.Repo, fname string, snapshot bool) error {
fi, err := os.Open(fname)
if err != nil {
return err
func ImportChain(r repo.Repo, fname string, snapshot bool) (err error) {
var rd io.Reader
var l int64
if strings.HasPrefix(fname, "http://") || strings.HasPrefix(fname, "https://") {
resp, err := http.Get(fname) //nolint:gosec
if err != nil {
return err
}
defer resp.Body.Close() //nolint:errcheck
if resp.StatusCode != http.StatusOK {
return xerrors.Errorf("non-200 response: %d", resp.StatusCode)
}
rd = resp.Body
l = resp.ContentLength
} else {
fname, err = homedir.Expand(fname)
if err != nil {
return err
}
fi, err := os.Open(fname)
if err != nil {
return err
}
defer fi.Close() //nolint:errcheck
st, err := os.Stat(fname)
if err != nil {
return err
}
rd = fi
l = st.Size()
}
defer fi.Close() //nolint:errcheck
lr, err := r.Lock(repo.FullNode)
if err != nil {
@ -353,8 +390,21 @@ func ImportChain(r repo.Repo, fname string, snapshot bool) error {
cst := store.NewChainStore(bs, mds, vm.Syscalls(ffiwrapper.ProofVerifier))
log.Info("importing chain from file...")
ts, err := cst.Import(fi)
log.Infof("importing chain from %s...", fname)
bufr := bufio.NewReaderSize(rd, 1<<20)
bar := pb.New64(l)
br := bar.NewProxyReader(bufr)
bar.ShowTimeLeft = true
bar.ShowPercent = true
bar.ShowSpeed = true
bar.Units = pb.U_BYTES
bar.Start()
ts, err := cst.Import(br)
bar.Finish()
if err != nil {
return xerrors.Errorf("importing chain failed: %w", err)
}

147
documentation/en/.glossary.json
View File

@ -1,146 +1 @@
{
"bellman": {
"title": "Bellman",
"value": "Bellman is a rust crate for building zk-SNARK circuits. It provides circuit traits and primitive structures, as well as basic gadget implementations such as booleans and number abstractions."
},
"nvme": {
"title": "NVMe",
"value": "(non-volatile memory express) is a host controller interface and storage protocol created to accelerate the transfer of data between enterprise and client systems and solid-state drives (SSDs) over a computer's high-speed Peripheral Component Interconnect Express (PCIe) bus."
},
"multiaddr": {
"title": "Multiaddr",
"value": "Multiaddr is a format for encoding addresses from various well-established network protocols. It is useful to write applications that future-proof their use of addresses, and allow multiple transport protocols and addresses to coexist."
},
"attofil": {
"title": "attoFIL",
"value": "AttoFIL is a word used to describe 10^-18 FIL. The word atto comes from the Norwegian and Danish term: atten eighteen."
},
"fil": {
"title": "FIL",
"value": "A ticker symbol is an abbreviation used to uniquely identify Filecoin when it is used in a wallet exchange or a cryptocurrency exchange."
},
"epost": {
"title": "Election Proof-of-Spacetime",
"value": "Election Proof-of-Spacetime couples the Proof-of-Spacetime process with block production, meaning that in order to produce a block, the miner must produce a valid Proof-of-Spacetime proof (snark output)."
},
"jwt": {
"title": "JWT",
"value": "JSON Web Tokens are an open, industry standard RFC 7519 method for representing claims securely between two parties."
},
"json-rpc": {
"title": "JSON-RPC",
"value": "JSON-RPC is a remote procedure call protocol encoded in JSON. It is a very simple protocol (and very similar to XML-RPC), defining only a few data types and commands."
},
"bls-address": {
"title": "BLS Signature (Address)",
"value": "A Boneh–Lynn–Shacham (BLS) signature is a digital signature scheme that allows a user to determine the authenticity of a signer, and is a commonly used signature scheme in the Filecoin Distributed Storage Network."
},
"faucet": {
"title": "Filecoin Test Faucet",
"value": "A webpage where you can get free test Filecoin to participate in the Testnet."
},
"chain": {
"title": "Chain",
"value": "The Filecoin Blockchain is a distributed virtual machine that achieves consensus, processes messages, accounts for storage, and maintains security in the Filecoin Protocol. It is the main interface linking various actors in the Filecoin system."
},
"miner-power": {
"title": "Miner Power",
"value": "Miner storage in relation to network storage, tracked in the power table."
},
"sector": {
"title": "Sector",
"value": "A fixed-size block of data of SECTOR_SIZE bytes which generally contains client's data."
},
"sealing": {
"title": "Sealing",
"value": "A slow encoding process that returns commitments and proofs for data being stored in a sector."
},
"seal": {
"title": "Seal",
"value": "A slow encoding process that returns commitments and proofs for data being stored in a sector."
},
"posts": {
"title": "Proof-of-Spacetime(s)",
"value": "Filecoin is a protocol token whose blockchain runs on a novel proof, called Proof-of-Spacetime, where blocks are created by miners that are storing data."
},
"filecoin-testnet": {
"title": "Filecoin Testnet",
"value": "Until we launch, we are making lots of changes to Lotus. The Testnet is expected to bring a few significant fixes/improvements. During Testnet, you can retrieve test filecoin from our network faucet to use as collateral to start mining. Test filecoin do not have any value – the official filecoin tokens will not be released until Mainnet launch."
},
"filecoin-decentralized-storage-market": {
"title": "Filecoin Decentralized Storage Market",
"value": "Storage Market subsystem is the data entry point into the network. Miners only earn power from data stored in a storage deal and all deals live on the Filecoin network."
},
"filecoin-proof-parameters": {
"title": "Filecoin Proof Parameters",
"value": "The proving algorithms rely on a large binary parameter file."
},
"lotus-devnet": {
"title": "DevNet",
"value": "On the DevNets, you can store data as a storage client and also try how Filecoin mining works. The devnets are an important development tool for those who anticipate building applications on top of the Filecoin protocol or storing data on the decentralized storage market. "
},
"filecoin-distributed-storage-network": {
"title": "Filecoin Distributed Storage Network",
"value": "Filecoin is a distributed storage network based on a blockchain mechanism. Filecoin miners can elect to provide storage capacity for the network, and thereby earn units of the Filecoin cryptocurrency (FIL) by periodically producing cryptographic proofs that certify that they are providing the capacity specified."
},
"lotus-node": {
"title": "Lotus Node",
"value": "The Lotus Node is full of capabilities. It runs the Blockchain system, makes retrieval deals, does data transfer, supports block producer logic, and syncs and validates the chain."
},
"block-rewards": {
"title": "Block Reward",
"value": "Over the entire lifetime of the protocol, 1,400,000,000 FIL (TotalIssuance) will be given out to miners. The rate at which the funds are given out is set to halve every six years, smoothly (not a fixed jump like in Bitcoin)."
},
"block-producer-miner": {
"title": "Miner (Block Producer)",
"value": "The Block Producer Miner's logic. It currently shares an interface and process with the Lotus Node. A Block Producer chooses which messages to include in a block and is rewarded according to each message’s gas price and consumption, forming a market."
},
"lotus-miner": {
"title": "Miner (lotus-miner)",
"value": "The Miner's logic. It has its own dedicated process. Contributes to the network through Sector commitments and Proofs of Spacetime to prove that it is storing the sectors it has commited to."
},
"swarm-port": {
"title": "Swarm Port (Libp2p)",
"value": "The LibP2P Swarm manages groups of connections to peers, handles incoming and outgoing streams, and is part of the miners implementation. The port value is part of the Host interface."
},
"daemon": {
"title": "Lotus Daemon",
"value": "A Daemon is a program that runs as a background process. A Daemon in the context of the Filecoin Distributed Storage Network may enable applications to communicate with peers, handle protocols, participate in pubsub, and interact with a distributed hash table (DHT)."
},
"storage-deal": {
"title": "Storage deal",
"value": "One of the two types of deals in Filecoin markets. Storage deals are recorded on the blockchain and enforced by the protocol."
},
"retrieval-deal": {
"title": "Retrieval deal",
"value": "One of the two types of deals in Filecoin markets. Retrieval deals are off chain and enabled by micropayment channel by transacting parties."
},
"deal-cid": {
"title": "Deal CID",
"value": "CID is a format for referencing content in distributed information systems, it is a way to store information so it can be retrieved based on its content, not its location. DealCID specifically is used in storage deals."
},
"data-cid": {
"title": "Data CID",
"value": "CID is a format for referencing content in distributed information systems, it is a way to store information so it can be retrieved based on its content, not its location. DataCID specifically is used to represent the file that is stored in the Filecoin Distributed Storage Network."
},
"cid": {
"title": "CID",
"value": "A CID is a self-describing content-addressed identifier. It uses cryptographic hashes to achieve content addressing. It uses several multiformats to achieve flexible self-description, namely multihash for hashes, multicodec for data content types, and multibase to encode the CID itself into strings."
},
"total-network-power": {
"title": "Total Network Power",
"value": "A reference to all the Power Tables for every subchain, accounting for each Lotus Miner on chain."
},
"chain-block-height": {
"title": "Chain Block Height",
"value": "Chain block height is defined as the number of blocks in the chain between any given block and the very first block in the blockchain."
},
"block-height": {
"title": "Block Height",
"value": "Height of the Merkle Tree of a sector. A sector is a contiguous array of bytes that a miner puts together, seals, and performs Proofs of Spacetime on."
},
"blocktime": {
"title": "Blocktime",
"value": "The time it takes for a Block to propagate to the whole network."
}
}
{}

231
documentation/en/.library.json
View File

@ -1,232 +1,25 @@
{
"posts": [
{
"title": "Hardware Requirements",
"slug": "en+hardware",
"github": "en/hardware.md",
"title": "About Lotus",
"slug": "",
"github": "en/about.md",
"value": null,
"posts": [
{
"title": "Testing Configuration",
"slug": "en+hardware-mining",
"github": "en/hardware-mining.md",
"value": null
}
]
"posts": []
},
{
"title": "Setup",
"slug": "en+getting-started",
"github": "en/getting-started.md",
"value": null,
"posts": [
{
"title": "Arch Linux Installation",
"slug": "en+install-lotus-arch",
"github": "en/install-lotus-arch.md",
"value": null
},
{
"title": "Ubuntu Installation",
"slug": "en+install-lotus-ubuntu",
"github": "en/install-lotus-ubuntu.md",
"value": null
},
{
"title": "Fedora Installation",
"slug": "en+install-lotus-fedora",
"github": "en/install-lotus-fedora.md",
"value": null
},
{
"title": "MacOS Installation",
"slug": "en+install-lotus-macos",
"github": "en/install-lotus-macos.md",
"value": null
},
{
"title": "Updating Lotus",
"slug": "en+updating-lotus",
"github": "en/updating-lotus.md",
"value": null
},
{
"title": "Join Testnet",
"slug": "en+join-testnet",
"github": "en/join-testnet.md",
"value": null
},
{
"title": "Use Lotus with systemd",
"slug": "en+install-systemd-services",
"github": "en/install-systemd-services.md",
"value": null
},
{
"title": "Setup Troubleshooting",
"slug": "en+setup-troubleshooting",
"github": "en/setup-troubleshooting.md",
"value": null
},
{
"title": "Environment Variables",
"slug": "en+env-vars",
"github": "en/environment-vars.md",
"value": null
}
]
},
{
"title": "Architecture",
"title": "Lotus Architecture (WIP)",
"slug": "en+arch",
"github": "en/architecture.md",
"github": "en/architecture/architecture.md",
"value": null,
"posts": [
{
"title": "The Message Pool",
"slug": "en+mpool",
"github": "en/mpool.md",
"value": null
}
{
"title": "The Message Pool",
"slug": "en+mpool",
"github": "en/architecture/mpool.md",
"value": null
}
]
},
{
"title": "Storage Mining",
"slug": "en+mining",
"github": "en/mining.md",
"value": null,
"posts": [
{
"title": "Lotus Worker",
"slug": "en+lotus-worker",
"github": "en/mining-lotus-worker.md",
"value": null
},
{
"title": "Static Ports",
"slug": "en+setting-a-static-port",
"github": "en/setting-a-static-port.md",
"value": null
},
{
"title": "Mining Troubleshooting",
"slug": "en+mining-troubleshooting",
"github": "en/mining-troubleshooting.md",
"value": null
}
]
},
{
"title": "Storing Data",
"slug": "en+storing-data",
"github": "en/storing-data.md",
"value": null,
"posts": [
{
"title": "Storage Troubleshooting",
"slug": "en+storing-data-troubleshooting",
"github": "en/storing-data-troubleshooting.md",
"value": null
},
{
"title": "Information for Miners",
"slug": "en+info-for-miners",
"github": "en/miner-deals.md",
"value": null
},
{
"title": "IPFS Integration",
"slug": "en+ipfs-client-integration",
"github": "en/storing-ipfs-integration.md",
"value": null
}
]
},
{
"title": "Retrieving Data",
"slug": "en+retrieving-data",
"github": "en/retrieving-data.md",
"value": null,
"posts": []
},
{
"title": "Payment Channels",
"slug": "en+payment-channels",
"github": "en/payment-channels.md",
"value": null,
"posts": []
},
{
"title": "Command Line Interface",
"slug": "en+cli",
"github": "en/cli.md",
"value": null,
"posts": []
},
{
"title": "API",
"slug": "en+api",
"github": "en/api.md",
"value": null,
"posts": [
{
"title": "Remote API Support",
"slug": "en+api-scripting-support",
"github": "en/api-scripting-support.md",
"value": null
},
{
"title": "API Methods",
"slug": "en+api-methods",
"github": "en/api-methods.md",
"value": null
},
{
"title": "API Troubleshooting",
"slug": "en+api-troubleshooting",
"github": "en/api-troubleshooting.md",
"value": null
}
]
},
{
"title": "Developer Tools",
"slug": "en+dev-tools",
"github": "en/dev-tools.md",
"value": null,
"posts": [
{
"title": "Setup Local Devnet",
"slug": "en+setup-local-dev-net",
"github": "en/local-dev-net.md",
"value": null,
"posts": []
},
{
"title": "Jaeger Tracing",
"slug": "en+dev-tools-jaeger-tracing",
"github": "en/dev-tools-jaeger-tracing.md",
"value": null,
"posts": []
}
]
},
{
"title": "FAQs",
"slug": "en+faqs",
"github": "en/faqs.md",
"value": null,
"posts": []
},
{
"title": "Glossary",
"slug": "en+glossary",
"github": "en/.glossary.json",
"value": null,
"custom": {
"glossary": true
},
"posts": []
}
]
}

16
documentation/en/README.md Normal file
View File

@ -0,0 +1,16 @@
# Lotus documentation
This folder contains some Lotus documentation mostly intended for Lotus developers.
User documentation (including documentation for miners) has been moved to specific Lotus sections in https://docs.filecoin.io:
- https://docs.filecoin.io/get-started/lotus
- https://docs.filecoin.io/store/lotus
- https://docs.filecoin.io/mine/lotus
- https://docs.filecoin.io/build/lotus
## The Lotu.sh site
The https://lotu.sh and https://docs.lotu.sh sites are generated from this folder based on the index provided by [.library.json](.library.json). This is done at the [lotus-docs repository](https://github.com/filecoin-project/lotus-docs), which contains Lotus as a git submodule.
To update the site, the lotus-docs repository should be updated with the desired version for the lotus git submodule. Once pushed to master, it will be auto-deployed.

0
documentation/en/dev/WIP-arch-complementary-notes.md → documentation/en/WIP-arch-complementary-notes.md
View File

19
documentation/en/about.md Normal file
View File

@ -0,0 +1,19 @@
# Lotus
Lotus is an implementation of the **Filecoin Distributed Storage Network**.
It is written in Go and provides a suite of command-line applications:
- Lotus Node (`lotus`): a Filecoin Node: validates network transactions, manages a FIL wallet, can perform storage and retrieval deals.
- Lotus Miner (`lotus-miner`): a Filecoin miner. See the the respective Lotus Miner section in the Mine documentation.
- Lotus Worker (`lotus-worker`): a worker that assists miners to perform mining-related tasks. See its respective guide for more information.
The [Lotus user documentation](https://docs.filecoin.io/get-started/lotus) is part of the [Filecoin documentation site](https://docs.filecoin.io):
* To install and get started with Lotus, visit the [Get Started section](https://docs.filecoin.io/get-started/lotus).
* Information about how to perform deals on the Filecoin network using Lotus can be found in the [Store section](https://docs.filecoin.io/store/lotus).
* Miners looking to provide storage to the Network can find the latest guides in the [Mine section](https://docs.filecoin.io/mine/lotus).
* Developers and integrators that wish to use the Lotus APIs can start in the [Build section](https://docs.filecoin.io/mine/lotus).
For more details about Filecoin, check out the [Filecoin Docs](https://docs.filecoin.io) and [Filecoin Spec](https://spec.filecoin.io/).

41
documentation/en/api-methods.md
View File

@ -169,6 +169,7 @@
* [SyncState](#SyncState)
* [SyncSubmitBlock](#SyncSubmitBlock)
* [SyncUnmarkBad](#SyncUnmarkBad)
* [SyncValidateTipset](#SyncValidateTipset)
* [Wallet](#Wallet)
* [WalletBalance](#WalletBalance)
* [WalletDefaultAddress](#WalletDefaultAddress)
@ -181,6 +182,7 @@
* [WalletSetDefault](#WalletSetDefault)
* [WalletSign](#WalletSign)
* [WalletSignMessage](#WalletSignMessage)
* [WalletValidateAddress](#WalletValidateAddress)
* [WalletVerify](#WalletVerify)
##
@ -3825,7 +3827,7 @@ Inputs:
]
```
Response: `2`
Response: `3`
### StateReadState
StateReadState returns the indicated actor's state.
@ -4379,6 +4381,28 @@ Inputs:
Response: `{}`
### SyncValidateTipset
SyncValidateTipset indicates whether the provided tipset is valid or not
Perms: read
Inputs:
```json
[
[
{
"/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"
},
{
"/": "bafy2bzacebp3shtrn43k7g3unredz7fxn4gj533d3o43tqn2p2ipxxhrvchve"
}
]
]
```
Response: `true`
## Wallet
@ -4585,6 +4609,21 @@ Response:
}
```
### WalletValidateAddress
WalletValidateAddress validates whether a given string can be decoded as a well-formed address
Perms: read
Inputs:
```json
[
"string value"
]
```
Response: `"t01234"`
### WalletVerify
WalletVerify takes an address, a signature, and some bytes, and indicates whether the signature is valid.
The address does not have to be in the wallet.

25
documentation/en/api-scripting-support.md
View File

@ -1,25 +0,0 @@
# Remote API Support
You may want to delegate the work **Lotus Miner** or **Lotus Node** performs to other machines.
Here is how to setup the necessary authorization and environment variables.
## Environment variables
Environmental variables are variables that are defined for the current shell and are inherited by any child shells or processes. Environmental variables are used to pass information into processes that are spawned from the shell.
Using the [JWT you generated](https://lotu.sh/en+api#how-do-i-generate-a-token-18865), you can assign it and the **multiaddr** to the appropriate environment variable.
```sh
# Lotus Node
FULLNODE_API_INFO="JWT_TOKEN:/ip4/127.0.0.1/tcp/1234/http"
# Lotus Miner
MINER_API_INFO="JWT_TOKEN:/ip4/127.0.0.1/tcp/2345/http"
```
You can also use `lotus auth api-info --perm admin` to quickly create _API_INFO env vars
- The **Lotus Node**'s `mutliaddr` is in `~/.lotus/api`.
- The default token is in `~/.lotus/token`.
- The **Lotus Miner**'s `multiaddr` is in `~/.lotusminer/config`.
- The default token is in `~/.lotusminer/token`.

36
documentation/en/api-troubleshooting.md
View File

@ -1,36 +0,0 @@
# API Troubleshooting
## Types: params
`params` must be an array. If there are no `params` you should still pass an empty array.
## Types: TipSet
For methods such as `Filecoin.StateMinerPower`, where the method accepts the argument of the type `TipSet`, you can pass `null` to use the current chain head.
```sh
curl -X POST \
-H "Content-Type: application/json" \
--data '{ "jsonrpc": "2.0", "method": "Filecoin.StateMinerPower", "params": ["t0101", null], "id": 3 }' \
'http://127.0.0.1:1234/rpc/v0'
```
## Types: Sending a CID
If you do not serialize the CID as a [JSON IPLD link](https://did-ipid.github.io/ipid-did-method/#txref), you will receive an error. Here is an example of a broken CURL request:
```sh
curl -X POST \
-H "Content-Type: application/json" \
--data '{ "jsonrpc": "2.0", "method":"Filecoin.ClientGetDealInfo", "params": ["bafyreiaxl446wlnu6t6dpq4ivrjf4gda4gvsoi4rr6mpxau7z25xvk5pl4"], "id": 0 }' \
'http://127.0.0.1:1234/rpc/v0'
```
To fix it, change the `params` property to:
```sh
curl -X POST \
-H "Content-Type: application/json" \
--data '{ "jsonrpc": "2.0", "method":"Filecoin.ClientGetDealInfo", "params": [{"/": "bafyreiaxl446wlnu6t6dpq4ivrjf4gda4gvsoi4rr6mpxau7z25xvk5pl4"}], "id": 0 }' \
'http://127.0.0.1:1234/rpc/v0'
```

85
documentation/en/api.md
View File

@ -1,85 +0,0 @@
# API
Here is an early overview of how to make API calls.
Implementation details for the **JSON-RPC** package are [here](https://github.com/filecoin-project/go-jsonrpc).
## Overview: How do you modify the config.toml to change the API endpoint?
API requests are made against `127.0.0.1:1234` unless you modify `.lotus/config.toml`.
Options:
- `http://[api:port]/rpc/v0` - HTTP endpoint
- `ws://[api:port]/rpc/v0` - Websocket endpoint
- `PUT http://[api:port]/rest/v0/import` - File import, it requires write permissions.
## What methods can I use?
For now, you can look into different files to find methods available to you based on your needs:
- [Both Lotus node + miner APIs](https://github.com/filecoin-project/lotus/blob/master/api/api_common.go)
- [Lotus node API](https://github.com/filecoin-project/lotus/blob/master/api/api_full.go)
- [Lotus miner API](https://github.com/filecoin-project/lotus/blob/master/api/api_storage.go)
The necessary permissions for each are in [api/struct.go](https://github.com/filecoin-project/lotus/blob/master/api/struct.go).
## How do I make an API request?
To demonstrate making an API request, we will take the method `ChainHead` from [api/api_full.go](https://github.com/filecoin-project/lotus/blob/master/api/api_full.go).
```go
ChainHead(context.Context) (*types.TipSet, error)
```
And create a CURL command. In this command, `ChainHead` is included as `{ "method": "Filecoin.ChainHead" }`:
```sh
curl -X POST \
-H "Content-Type: application/json" \
--data '{ "jsonrpc": "2.0", "method": "Filecoin.ChainHead", "params": [], "id": 3 }' \
'http://127.0.0.1:1234/rpc/v0'
```
If the request requires authorization, add an authorization header:
```sh
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(cat ~/.lotusminer/token)" \
--data '{ "jsonrpc": "2.0", "method": "Filecoin.ChainHead", "params": [], "id": 3 }' \
'http://127.0.0.1:1234/rpc/v0'
```
> In the future we will add a playground to make it easier to build and experiment with API requests.
## CURL authorization
To authorize your request, you will need to include the **JWT** in a HTTP header, for example:
```sh
-H "Authorization: Bearer $(cat ~/.lotusminer/token)"
```
Admin token is stored in `~/.lotus/token` for the **Lotus Node** or `~/.lotusminer/token` for the **Lotus Miner**.
## How do I generate a token?
To generate a JWT with custom permissions, use this command:
```sh
# Lotus Node
lotus auth create-token --perm admin
# Lotus Miner
lotus-miner auth create-token --perm admin
```
## What authorization level should I use?
When viewing [api/apistruct/struct.go](https://github.com/filecoin-project/lotus/blob/master/api/apistruct/struct.go), you will encounter these types:
- `read` - Read node state, no private data.
- `write` - Write to local store / chain, and `read` permissions.
- `sign` - Use private keys stored in wallet for signing, `read` and `write` permissions.
- `admin` - Manage permissions, `read`, `write`, and `sign` permissions.

2
documentation/en/architecture.md → documentation/en/architecture/architecture.md
View File

@ -6,7 +6,7 @@ Filecoin protocol, validating the blocks and state transitions.
The specification for the Filecoin protocol can be found [here](https://filecoin-project.github.io/specs/).
For information on how to setup and operate a Lotus node,
please follow the instructions [here](https://lotu.sh/en+getting-started).
please follow the instructions [here](en+getting-started).
# Components

0
documentation/en/mpool.md → documentation/en/architecture/mpool.md
View File

108
documentation/en/cli.md
View File

@ -1,108 +0,0 @@
# Lotus Command Line Interface
The Command Line Interface (CLI) is a convenient way to interact with
a Lotus node. You can use the CLI to operate your node,
get information about the blockchain,
manage your accounts and transfer funds,
create storage deals, and much more!
The CLI is intended to be self-documenting, so when in doubt, simply add `--help`
to whatever command you're trying to run! This will also display all of the
input parameters that can be provided to a command.
We highlight some of the commonly
used features of the CLI below.
All CLI commands should be run from the home directory of the Lotus project.
## Operating a Lotus node
### Starting up a node
```sh
lotus daemon
```
This command will start up your Lotus node, with its API port open at 1234.
You can pass `--api=<number>` to use a different port.
### Checking your sync progress
```sh
lotus sync status
```
This command will print your current tipset height under `Height`, and the target tipset height
under `Taregt`.
You can also run `lotus sync wait` to get constant updates on your sync progress.
### Getting the head tipset
```sh
lotus chain head
```
### Control the logging level
```sh
lotus log set-level
```
This command can be used to toggle the logging levels of the different
systems of a Lotus node. In decreasing order
of logging detail, the levels are `debug`, `info`, `warn`, and `error`.
As an example,
to set the `chain` and `blocksync` to log at the `debug` level, run
`lotus log set-level --system chain --system blocksync debug`.
To see the various logging system, run `lotus log list`.
### Find out what version of Lotus you're running
```sh
lotus version
```
## Managing your accounts
### Listing accounts in your wallet
```sh
lotus wallet list
```
### Creating a new account
```sh
lotus wallet new bls
```
This command will create a new BLS account in your wallet; these
addresses start with the prefix `t3`. Running `lotus wallet new secp256k1`
(or just `lotus wallet new`) will create
a new Secp256k1 account, which begins with the prefix `t1`.
### Getting an account's balance
```sh
lotus wallet balance <address>
```
### Transferring funds
```sh
lotus send --source=<source address> <destination address> <amount>
```
This command will transfer `amount` (in attoFIL) from `source address` to `destination address`.
### Importing an account into your wallet
```sh
lotus wallet import <path to private key>
```
This command will import an account whose private key is saved at the specified file.
### Exporting an account from your wallet
```sh
lotus wallet export <address>
```
This command will print out the private key of the specified address
if it is in your wallet. Always be careful with your private key!

0
documentation/en/dev/create-miner.md → documentation/en/create-miner.md
View File

3
documentation/en/dev-tools.md
View File

@ -1,3 +0,0 @@
# Developer Tools
> Running a local network can be a great way to understand how Lotus works and test your setup.

65
documentation/en/environment-vars.md
View File

@ -1,65 +0,0 @@
# Lotus Environment Variables
## Building
## Common
The environment variables are common across most lotus binaries.
### `LOTUS_FD_MAX`
Sets the file descriptor limit for the process. This should be set high (8192
or higher) if you ever notice 'too many open file descriptor' errors.
### `LOTUS_JAEGER`
This can be set to enable jaeger trace reporting. The value should be the url
of the jaeger trace collector, the default for most jaeger setups should be
`localhost:6831`.
### `LOTUS_DEV`
If set to a non-empty value, certain parts of the application will print more
verbose information to aid in development of the software. Not recommended for
end users.
## Lotus Daemon
### `LOTUS_PATH`
Sets the location for the lotus daemon on-disk repo. If left empty, this defaults to `~/.lotus`.
### `LOTUS_SKIP_GENESIS_CHECK`
Can be set to `_yes_` if you wish to run a lotus network with a different
genesis than the default one built into your lotus binary.
### `LOTUS_CHAIN_TIPSET_CACHE`
Sets the cache size for the chainstore tipset cache. The default value is 8192,
but if your usage of the lotus API involves frequent arbitrary tipset lookups,
you may want to increase this.
### `LOTUS_CHAIN_INDEX_CACHE`
Sets the cache size for the chainstore epoch index cache. The default value is 32768,
but if your usage of the lotus API involves frequent deep chain lookups for
block heights that are very far from the current chain height, you may want to
increase this.
### `LOTUS_BSYNC_MSG_WINDOW`
Set the initial maximum window size for message fetching blocksync requests. If
you have a slower internet connection and are having trouble syncing, you might
try lowering this down to 10-20 for a 'poor' internet connection.
## Lotus Miner
A number of environment variables are respected for configuring the behavior of the filecoin proving subsystem. For more details on those [see here](https://github.com/filecoin-project/rust-fil-proofs/#settings).
### `LOTUS_MINER_PATH`
Sets the location for the lotus miners on-disk repo. If left empty, this defaults to `~/.lotusminer`.

138
documentation/en/faqs.md
View File

@ -1,138 +0,0 @@
# Frequently Asked Questions
Here are some FAQs concerning the Lotus implementation and participation in
Testnet.
For questions concerning the broader Filecoin project, please
go [here](https://filecoin.io/faqs/).
## Introduction to Lotus
### What is Lotus?
Lotus is an implementation of the **Filecoin Distributed Storage Network**, written in Go.
It is designed to be modular and interoperable with any other implementation of the Filecoin Protocol.
More information about Lotus can be found [here](https://lotu.sh/).
### What are the components of Lotus?
Lotus is composed of two separate pieces that can talk to each other:
The Lotus Node can sync the blockchain, validating all blocks, transfers, and deals
along the way. It can also facilitate the creation of new storage deals. If you are not
interested in providing your own storage to the network, and do not want to produce blocks
yourself, then the Lotus Node is all you need!
The Lotus Miner does everything you need for the registration of storage, and the
production of new blocks. The Lotus Miner communicates with the network by talking
to a Lotus Node over the JSON-RPC API.
## Setting up a Lotus Node
### How do I set up a Lotus Node?
Follow the instructions found [here](https://lotu.sh/en+getting-started).
### Where can I get the latest version of Lotus?
Download the binary tagged as the `Latest Release` from the
[Lotus Github repo](https://github.com/filecoin-project/lotus/releases).
### What operating systems can Lotus run on?
Lotus can build and run on most Linux and MacOS systems with at least
8GB of RAM. Windows is not yet supported.
### How can I update to the latest version of Lotus?
To update Lotus, follow the instructions [here](https://lotu.sh/en+updating-lotus).
### How do I prepare a fresh installation of Lotus?
Stop the Lotus daemon, and delete all related files, including sealed and chain data by
running `rm ~/.lotus ~/.lotusminer`.
Then, install Lotus afresh by following the instructions
found [here](https://lotu.sh/en+getting-started).
### Can I configure where the node's config and data goes?
Yes! The `LOTUS_PATH` variable sets the path for where the Lotus node's data is written.
The `LOTUS_MINER_PATH` variable does the same for miner-specific information.
## Interacting with a Lotus Node
### How can I communicate with a Lotus Node?
Lotus Nodes have a command-line interface, as well as a JSON-RPC API.
### What are the commands I can send using the command-line interface?
The command-line interface is self-documenting, try running `lotus --help` from the `lotus` home
directory for more.
### How can I send a request over the JSON-RPC API?
Information on how to send a `cURL` request to the JSON-RPC API can be found
[here](https://lotu.sh/en+api). A JavaScript client is under development.
### What are the requests I can send over the JSON-RPC API?
Please have a look at the
[source code](https://github.com/filecoin-project/lotus/blob/master/api/api_full.go)
for a list of methods supported by the JSON-RPC API.
## The Test Network
### What is Testnet?
Testnet is a live network of Lotus Nodes run by the
community for testing purposes.
It has 2 PiB of storage (and growing!) dedicated to it.
### Is FIL on the Testnet worth anything?
Nothing at all! Real-world incentives may be provided in a future phase of Testnet, but this is
yet to be confirmed.
### How can I see the status of Testnet?
The [dashboard](https://stats.testnet.filecoin.io/) displays the status of the network as
well as a ton
of other metrics you might find interesting.
## Mining with a Lotus Node on Testnet
### How do I get started mining with Lotus?
Follow the instructions found [here](https://lotu.sh/en+mining).
### What are the minimum hardware requirements?
An example test configuration, and minimum hardware requirements can be found
[here](https://lotu.sh/en+hardware-mining).
Note that these might NOT be the minimum requirements for mining on Mainnet.
### What are some GPUs that have been tested?
A list of benchmarked GPUs can be found [here](https://lotu.sh/en+hardware-mining#benchmarked-gpus-7393).
### Why is my GPU not being used when sealing a sector?
Sealing a sector does not involve constant GPU operations. It's possible
that your GPU simply isn't necessary at the moment you checked.
## Advanced questions
### Is there a Docker image for lotus?
Community-contributed Docker and Docker Compose examples are available
[here](https://github.com/filecoin-project/lotus/tree/master/tools/dockers/docker-examples).
### How can I run two miners on the same machine?
You can do so by changing the storage path variable for the second miner, e.g.,
`LOTUS_MINER_PATH=~/.lotusminer2`. You will also need to make sure that no ports collide.
### How do I setup my own local devnet?
Follow the instructions found [here](https://lotu.sh/en+setup-local-dev-net).

23
documentation/en/getting-started.md
View File

@ -1,23 +0,0 @@
# Lotus
Lotus is an implementation of the **Filecoin Distributed Storage Network**. You can run the Lotus software client to join the **Filecoin Testnet**.
For more details about Filecoin, check out the [Filecoin Docs](https://docs.filecoin.io) and [Filecoin Spec](https://filecoin-project.github.io/specs/).
## What can I learn here?
- How to install Lotus on [Arch Linux](https://lotu.sh/en+install-lotus-arch), [Ubuntu](https://lotu.sh/en+install-lotus-ubuntu), or [MacOS](https://lotu.sh/en+install-lotus-macos).
- Joining the [Lotus Testnet](https://lotu.sh/en+join-testnet).
- [Storing](https://lotu.sh/en+storing-data) or [retrieving](https://lotu.sh/en+retrieving-data) data.
- Mining Filecoin using the **Lotus Miner** in your [CLI](https://lotu.sh/en+mining).
## How is Lotus designed?
Lotus is architected modularly to keep clean API boundaries while using the same process. Installing Lotus will include two separate programs:
- The **Lotus Node**
- The **Lotus Miner**
The **Lotus Miner** is intended to be run on the machine that manages a single miner instance, and is meant to communicate with the **Lotus Node** via the websocket **JSON-RPC** API for all of the chain interaction needs.
This way, a mining operation may easily run a **Lotus Miner** or many of them, connected to one or many **Lotus Node** instances.

54
documentation/en/hardware-mining.md
View File

@ -1,54 +0,0 @@
# Protocol Labs Standard Testing Configuration
> This documentation page describes the standard testing configuration the Protocol Labs team has used to test **Lotus Miner**s on Lotus. There is no guarantee this testing configuration will be suitable for Filecoin storage mining at MainNet launch. If you need to buy new hardware to join the Filecoin Testnet, we recommend to buy no more hardware than you require for testing. To learn more please read this [Protocol Labs Standard Testing Configuration post](https://filecoin.io/blog/filecoin-testnet-mining/).
**Sector sizes** and **minimum pledged storage** required to mine blocks are two very important Filecoin Testnet parameters that impact hardware decisions. We will continue to refine all parameters during Testnet.
BECAUSE OF THIS, OUR STANDARD TESTING CONFIGURATION FOR FILECOIN MAINNET CAN AND WILL CHANGE. YOU HAVE BEEN WARNED.
## Example configuration
The setup below is a minimal example for sealing 32 GiB sectors on Lotus:
- 2 TB of hard drive space.
- 8 core CPU
- 128 GiB of RAM
Note that 1GB sectors don't require as high of specs, but are likely to be removed as we improve the performance of 32GB sector sealing.
For the first part of the sealing process, AMD CPU's are __highly recommended__, because of the `Intel SHA Extensions` instruction set that is available there ever since the `Zen` microarchitecture. Hence, AMD CPU's seem to perform much better on the testnet than other CPU's. Contrary to what the name implies, this extended instruction set is not available on recent Intel desktop/server chips.
## Testnet discoveries
- If you only have 128GiB of ram, enabling 256GB of **NVMe** swap on an SSD will help you avoid out-of-memory issues while mining.
## Benchmarked GPUs
GPUs are a must for getting **block rewards**. Here are a few that have been confirmed to generate **SNARKs** quickly enough to successfully mine blocks on the Lotus Testnet.
- GeForce RTX 2080 Ti
- GeForce RTX 2080 SUPER
- GeForce RTX 2080
- GeForce GTX 1080 Ti
- GeForce GTX 1080
- GeForce GTX 1060
## Testing other GPUs
If you want to test a GPU that is not explicitly supported, use the following global **environment variable**:
```sh
BELLMAN_CUSTOM_GPU="<NAME>:<NUMBER_OF_CORES>"
```
Here is an example of trying a GeForce GTX 1660 Ti with 1536 cores.
```sh
BELLMAN_CUSTOM_GPU="GeForce GTX 1660 Ti:1536"
```
To get the number of cores for your GPU, you will need to check your card’s specifications.
## Benchmarking
Here is a [benchmarking tool](https://github.com/filecoin-project/lotus/tree/master/cmd/lotus-bench) and a [GitHub issue thread](https://github.com/filecoin-project/lotus/issues/694) for those who wish to experiment with and contribute hardware setups for the **Filecoin Testnet**.

7
documentation/en/hardware.md
View File

@ -1,7 +0,0 @@
# Hardware
> This page is a work in progress. Exact mining requirements are still in the works.
Lotus can build and run on most [Linux](https://ubuntu.com/) and [MacOS](https://www.apple.com/macos) systems with at least 8GiB of RAM.
Windows is not yet supported.

51
documentation/en/install-lotus-arch.md
View File

@ -1,51 +0,0 @@
# Arch Linux Instructions
These steps will install the following dependencies:
- go (1.14 or higher)
- gcc (7.4.0 or higher)
- git (version 2 or higher)
- bzr (some go dependency needs this)
- jq
- pkg-config
- opencl-icd-loader
- opencl driver (like nvidia-opencl on arch) (for GPU acceleration)
- opencl-headers (build)
- rustup (proofs build)
- llvm (proofs build)
- clang (proofs build)
### Install dependencies
```sh
sudo pacman -Syu opencl-icd-loader gcc git bzr jq pkg-config opencl-icd-loader opencl-headers
```
### Install Go 1.14
Install the latest version of Go by following [the docs on their website](https://golang.org/doc/install).
### Clone the Lotus repository
```sh
git clone https://github.com/filecoin-project/lotus.git
cd lotus/
```
### Build the Lotus binaries from source and install
! **If you are running an AMD platform or if your CPU supports SHA extensions you will want to build the Filecoin proofs natively**
```sh
make clean && make all
sudo make install
```
#### Native Filecoin FFI building
```sh
env env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps all
sudo make install
```
After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

54
documentation/en/install-lotus-fedora.md
View File

@ -1,54 +0,0 @@
# Fedora Instructions
> tested on 30
**NOTE:** If you have an AMD GPU the opencl instructions may be incorrect...
These steps will install the following dependencies:
- go (1.14 or higher)
- gcc (7.4.0 or higher)
- git (version 2 or higher)
- bzr (some go dependency needs this)
- jq
- pkg-config
- rustup (proofs build)
- llvm (proofs build)
- clang (proofs build)
### Install dependencies
```sh
$ sudo dnf -y update
$ sudo dnf -y install gcc git bzr jq pkgconfig mesa-libOpenCL mesa-libOpenCL-devel opencl-headers ocl-icd ocl-icd-devel clang llvm
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
### Install Go 1.14
Install the latest version of Go by following [the docs on their website](https://golang.org/doc/install).
### Clone the Lotus repository
```sh
git clone https://github.com/filecoin-project/lotus.git
cd lotus/
```
### Build the Lotus binaries from source and install
! **If you are running an AMD platform or if your CPU supports SHA extensions you will want to build the Filecoin proofs natively**
```sh
$ make clean && make all
$ sudo make install
```
#### Native Filecoin FFI building
```sh
env env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps all
sudo make install
```
After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus TestNet](https://lotu.sh/en+join-testnet).

62
documentation/en/install-lotus-macos.md
View File

@ -1,62 +0,0 @@
# MacOS Instructions
## Get XCode Command Line Tools
To check if you already have the XCode Command Line Tools installed via the CLI, run:
```sh
xcode-select -p
```
If this command returns a path, you can move on to the next step. Otherwise, to install via the CLI, run:
```sh
xcode-select --install
```
To update, run:
```sh
sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install
```
## Get HomeBrew
We recommend that MacOS users use [HomeBrew](https://brew.sh) to install each the necessary packages.
Check if you have HomeBrew:
```sh
brew -v
```
This command returns a version number if you have HomeBrew installed and nothing otherwise.
In your terminal, enter this command to install Homebrew:
```sh
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
Use the command `brew install` to install the following packages:
```sh
brew install go bzr jq pkg-config rustup
```
Clone
```sh
git clone https://github.com/filecoin-project/lotus.git
cd lotus/
```
Build
```sh
make clean && make all
sudo make install
```
After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

54
documentation/en/install-lotus-ubuntu.md
View File

@ -1,54 +0,0 @@
# Ubuntu Instructions
These steps will install the following dependencies:
- go (1.14 or higher)
- gcc (7.4.0 or higher)
- git (version 2 or higher)
- bzr (some go dependency needs this)
- jq
- pkg-config
- opencl-icd-loader
- opencl driver (like nvidia-opencl on arch) (for GPU acceleration)
- opencl-headers (build)
- rustup (proofs build)
- llvm (proofs build)
- clang (proofs build)
### Install dependencies
```sh
sudo apt update
sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl
sudo apt upgrade
```
### Install Go 1.14
Install the latest version of Go by following [the docs on their website](https://golang.org/doc/install).
### Clone the Lotus repository
```sh
git clone https://github.com/filecoin-project/lotus.git
cd lotus/
```
### Build the Lotus binaries from source and install
! **If you are running an AMD platform or if your CPU supports SHA extensions you will want to build the Filecoin proofs natively**
```sh
make clean && make all
sudo make install
```
#### Native Filecoin FFI building
```sh
env env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps all
sudo make install
```
After installing Lotus, you can run the `lotus` command directly from your CLI to see usage documentation. Next, you can join the [Lotus Testnet](https://lotu.sh/en+join-testnet).

145
documentation/en/install-systemd-services.md
View File

@ -1,145 +0,0 @@
# Use Lotus with systemd
Lotus is capable of running as a systemd service daemon. You can find installable service files for systemd in the [lotus repo scripts directory](https://github.com/filecoin-project/lotus/tree/master/scripts) as files with `.service` extension. In order to install these service files, you can copy these `.service` files to the default systemd unit load path.
The services expect their binaries to be present in `/usr/local/bin/`. You can use `make` to install them by running:
```sh
$ sudo make install
```
for `lotus` and `lotus-miner` and
```sh
$ sudo make install-chainwatch
```
for the `chainwatch` tool.
## Installing services via `make`
If your host uses the default systemd unit load path, the `lotus-daemon` and `lotus-miner` services can be installed by running:
```sh
$ sudo make install-services
```
To install the the `lotus-chainwatch` service run:
```sh
$ sudo make install-chainwatch-service
```
You can install all services together by running:
```sh
$ sudo make install-all-services
```
The `lotus-daemon` and the `lotus-miner` services can be installed individually too by running:
```sh
$ sudo make install-daemon-service
```
and
```sh
$ sudo make install-miner-service
```
### Notes
When installing the `lotus-miner` and/or `lotus-chainwatch` service the `lotus-daemon` service gets automatically installed since the other two services depend on it being installed to run.
All `install-*-service*` commands will install the latest binaries in the lotus build folders to `/usr/local/bin/`. If you do not want to use the latest build binaries please copy the `*.service` files by hand.
## Removing via `make`
All services can beremoved via `make`. To remove all services together run:
```sh
$ sudo make clean-all-services
```
Individual services can be removed by running:
```sh
$ sudo make clean-chainwatch-services
$ sudo make clean-miner-services
$ sudo make clean-daemon-services
```
### Notes
The services will be stoppend and disabled when removed.
Removing the `lotus-daemon` service will automatically remove the depending services `lotus-miner` and `lotus-chainwatch`.
## Controlling services
All service can be controlled with the `systemctl`. A few basic control commands are listed below. To get detailed infos about the capabilities of the `systemctl` command please consult your distributions man pages by running:
```sh
$ man systemctl
```
### Start/Stop services
You can start the services by running:
```sh
$ sudo systemctl start lotus-daemon
$ sudo systemctl start lotus-miner
$ sudo systemctl start lotus-chainwatch
```
and can be stopped by running:
```sh
$ sudo systemctl stop lotus-daemon
$ sudo systemctl stop lotus-miner
$ sudo systemctl stop lotus-chainwatch
```
### Enabling services on startup
To enable the services to run automatically on startup execute:
```sh
$ sudo systemctl enable lotus-daemon
$ sudo systemctl enable lotus-miner
$ sudo systemctl enable lotus-chainwatch
```
To disable the services on startup run:
```sh
$ sudo systemctl disable lotus-daemon
$ sudo systemctl disable lotus-miner
$ sudo systemctl disable lotus-chainwatch
```
### Notes
Systemd will not let services be enabled or started without their requirements. Starting the `lotus-chainwatch` and/or `lotus-miner` service with automatically start the `lotus-daemon` service (if installed!). Stopping the `lotus-daemon` service will stop the other two services. The same pattern is executed for enabling and disabling the services.
## Interacting with service logs
Logs from the services can be reviewed using `journalctl`.
### Follow logs from a specific service unit
```sh
$ sudo journalctl -u lotus-daemon -f
```
### View logs in reverse order
```sh
$ sudo journalctl -u lotus-miner -r
```
### Log files
Besides the systemd service logs all services save their own log files in `/var/log/lotus/`.

0
documentation/en/dev-tools-jaeger-tracing.md → documentation/en/jaeger-tracing.md
View File

93
documentation/en/join-testnet.md
View File

@ -1,93 +0,0 @@
# Join Testnet
## Introduction
Anyone can set up a **Lotus Node** and connect to the **Lotus Testnet**. This is the best way to explore the current CLI and the **Filecoin Decentralized Storage Market**.
## Note: Using the Lotus Node from China
If you are trying to use `lotus` from China. You should set this **environment variable** on your machine:
```sh
export IPFS_GATEWAY="https://proof-parameters.s3.cn-south-1.jdcloud-oss.com/ipfs/"
```
## Get started
Start the **daemon** using the default configuration in `./build`:
```sh
lotus daemon
```
In another terminal window, check your connection with peers:
```sh
lotus net peers | wc -l
```
In order to connect to the network, you need to be connected to at least 1 peer. If you’re seeing 0 peers, read our [troubleshooting notes](https://lotu.sh/en+setup-troubleshooting).
Make sure that you have a reasonable "open files limit" set on your machine, such as 10000. If you're seeing a lower value, such as 256 (default on macOS), read our [troubleshooting notes](https://lotu.sh/en+setup-troubleshooting) on how to update it prior to starting the Lotus daemon.
## Chain sync
While the daemon is running, the next requirement is to sync the chain. Run the command below to view the chain sync progress. To see current chain height, visit the [network stats page](https://stats.testnet.filecoin.io/).
```sh
lotus sync wait
```
- This step will take anywhere between a few hours to a couple of days.
- You will be able to perform **Lotus Testnet** operations after it is finished.
## Create your first address
Initialize a new wallet:
```sh
lotus wallet new
```
Sometimes your operating system may limit file name length to under 150 characters. You need to use a file system that supports long filenames.
Here is an example of the response:
```sh
t1aswwvjsae63tcrniz6x5ykvsuotlgkvlulnqpsi
```
- Visit the [faucet](http://spacerace.faucet.glif.io/) to add funds.
- Paste the address you created under REQUEST.
- Press the Request button.
## Check wallet address balance
Wallet balances in the Lotus Testnet are in **FIL**, the smallest denomination of FIL is an **attoFil**, where 1 attoFil = 10^-18 FIL.
```sh
lotus wallet balance <YOUR_NEW_ADDRESS>
```
You will not see any attoFIL in your wallet if your **chain** is not fully synced.
## Send FIL to another wallet
To send FIL to another wallet from your default account, use this command:
```
lotus send <target> <amount>
```
## Configure your node's connectivity
To effectively accept incoming storage & retrieval deals, your Lotus node needs to be accessible to other nodes on the network. To improve your connectivity, be sure to:
- [Set the multiaddresses for you miner to listen on](https://docs.filecoin.io/mine/connectivity/#setting-multiaddresses)
- [Maintain a healthy peer count](https://docs.filecoin.io/mine/connectivity/#checking-peer-count)
- [Enable port forwarding](https://docs.filecoin.io/mine/connectivity/#port-forwarding)
- [Configure your public IP address and port](https://docs.filecoin.io/mine/connectivity/#setting-a-public-ip-address)
## Monitor the dashboard
To see the latest network activity, including **chain block height**, **block height**, **blocktime**, **total network power**, largest **block producer miner**, check out the [monitoring dashboard](https://stats.testnet.filecoin.io).

54
documentation/en/local-dev-net.md
View File

@ -1,54 +0,0 @@
# Setup Local Devnet
Build the Lotus Binaries in debug mode, This enables the use of 2048 byte sectors.
```sh
make 2k
```
Set the `LOTUS_SKIP_GENESIS_CHECK` environment variable to `_yes_`. This tells your
Lotus node that it's okay if the genesis being used doesn't match any baked-in
genesis.
```sh
export LOTUS_SKIP_GENESIS_CHECK=_yes_
```
Download the 2048 byte parameters:
```sh
./lotus fetch-params 2048
```
Pre-seal some sectors:
```sh
./lotus-seed pre-seal --sector-size 2KiB --num-sectors 2
```
Create the genesis block and start up the first node:
```sh
./lotus-seed genesis new localnet.json
./lotus-seed genesis add-miner localnet.json ~/.genesis-sectors/pre-seal-t01000.json
./lotus daemon --lotus-make-genesis=devgen.car --genesis-template=localnet.json --bootstrap=false
```
Then, in another console, import the genesis miner key:
```sh
./lotus wallet import --as-default ~/.genesis-sectors/pre-seal-t01000.key
```
Set up the genesis miner:
```sh
./lotus-miner init --genesis-miner --actor=t01000 --sector-size=2KiB --pre-sealed-sectors=~/.genesis-sectors --pre-sealed-metadata=~/.genesis-sectors/pre-seal-t01000.json --nosync
```
Now, finally, start up the miner:
```sh
./lotus-miner run --nosync
```
If all went well, you will have your own local Lotus Devnet running.

39
documentation/en/miner-deals.md
View File

@ -1,39 +0,0 @@
# Information for Miners
Here is how a miner can get set up to accept storage deals. The first step is
to install a Lotus node and sync to the top of the chain.
## Set up an ask
```
lotus-miner set-price <price>
```
This command will set up your miner to accept deal proposals that meet the input price.
The price is inputted in FIL per GiB per epoch, and the default is 0.0000000005.
## Ensure you can be discovered
Clients need to be able to find you in order to make storage deals with you.
While there isn't necessarily anything you need to do to become discoverable, here are some things you can
try to check that people can connect to you.
To start off, make sure you are connected to at least some peers, and your port is
open and working.
### Connect to your own node
If you are in contact with someone else running Lotus, you can ask them to try connecting
to your node. To do so, provide them your peer ID, which you can get by running `lotus net id` on
your node.
They can then try running `lotus net findpeer <peerID>` to get your address(es), and can then
run `lotus net connect <address>` to connect to you. If successful, your node will now
appear on their peers list (run `lotus net peers` to check).
You can also check this by running a second instance of Lotus yourself.
### Query your own ask
A client should be able to find your ask by running `lotus client query-ask <minerID>`. If
someone is not able to retrieve your ask by doing so, then there is an issue with your node.

81
documentation/en/mining-lotus-worker.md
View File

@ -1,81 +0,0 @@
# Lotus Worker
The **Lotus Worker** is an extra process that can offload heavy processing tasks from your **Lotus Miner**. The sealing process automatically runs in the **Lotus Miner** process, but you can use the Worker on another machine communicating over a fast network to free up resources on the machine running the mining process.
## Note: Using the Lotus Worker from China
If you are trying to use `lotus-worker` from China. You should set this **environment variable** on your machine:
```sh
export IPFS_GATEWAY="https://proof-parameters.s3.cn-south-1.jdcloud-oss.com/ipfs/"
```
## Get Started
Make sure that the `lotus-worker` is compiled and installed by running:
```sh
make lotus-worker
```
## Setting up the Miner
First, you will need to ensure your `lotus-miner`'s API is accessible over the network.
To do this, open up `~/.lotusminer/config.toml` (Or if you manually set `LOTUS_MINER_PATH`, look under that directory) and look for the API field.
Default config:
```toml
[API]
ListenAddress = "/ip4/127.0.0.1/tcp/2345/http"
RemoteListenAddress = "127.0.0.1:2345"
```
To make your node accessible over the local area network, you will need to determine your machines IP on the LAN, and change the `127.0.0.1` in the file to that address.
A more permissive and less secure option is to change it to `0.0.0.0`. This will allow anyone who can connect to your computer on that port to access the [API](https://lotu.sh/en+api). They will still need an auth token.
`RemoteListenAddress` must be set to an address which other nodes on your network will be able to reach.
Next, you will need to [create an authentication token](https://lotu.sh/en+api-scripting-support#generate-a-jwt-46). All Lotus APIs require authentication tokens to ensure your processes are as secure against attackers attempting to make unauthenticated requests to them.
### Connect the Lotus Worker
On the machine that will run `lotus-worker`, set the `MINER_API_INFO` environment variable to `TOKEN:MINER_NODE_MULTIADDR`. Where `TOKEN` is the token we created above, and `NIMER_NODE_MULTIADDR` is the `multiaddr` of the **Lotus Miner** API that was set in `config.toml`.
Once this is set, run:
```sh
lotus-worker run
```
If you are running multiple workers on the same host, you will need to specify the `--listen` flag and ensure each worker is on a different port.
To check that the **Lotus Worker** is connected to your **Lotus Miner**, run `lotus-miner sealing workers` and check that the remote worker count has increased.
```sh
why@computer ~/lotus> lotus-miner sealing workers
Worker 0, host computer
CPU: [ ] 0 core(s) in use
RAM: [|||||||||||||||||| ] 28% 18.1 GiB/62.7 GiB
VMEM: [|||||||||||||||||| ] 28% 18.1 GiB/62.7 GiB
GPU: GeForce RTX 2080, not used
Worker 1, host othercomputer
CPU: [ ] 0 core(s) in use
RAM: [|||||||||||||| ] 23% 14 GiB/62.7 GiB
VMEM: [|||||||||||||| ] 23% 14 GiB/62.7 GiB
GPU: GeForce RTX 2080, not used
```
### Running locally for manually managing process priority
You can also run the **Lotus Worker** on the same machine as your **Lotus Miner**, so you can manually manage the process priority.
To do so you have to first __disable all seal task types__ in the miner config. This is important to prevent conflicts between the two processes.
You can then run the miner on your local-loopback interface;
```sh
lotus-worker run
```

66
documentation/en/mining-troubleshooting.md
View File

@ -1,66 +0,0 @@
# Mining Troubleshooting
## Config: Filecoin Proof Parameters directory
If you want to put the **Filecoin Proof Parameters** in a different directory, use the following environment variable:
```sh
FIL_PROOFS_PARAMETER_CACHE
```
## Error: Can't acquire bellman.lock
The **Bellman** lockfile is created to lock a GPU for a process. This bug can occur when this file isn't properly cleaned up:
```sh
mining block failed: computing election proof: github.com/filecoin-project/lotus/miner.(*Miner).mineOne
```
This bug occurs when the miner can't acquire the `bellman.lock`. To fix it you need to stop the `lotus-miner` and remove `/tmp/bellman.lock`.
## Error: Failed to get api endpoint
```sh
lotus-miner info
# WARN main lotus-miner/main.go:73 failed to get api endpoint: (/Users/myrmidon/.lotusminer) %!w(*errors.errorString=&{API not running (no endpoint)}):
```
If you see this, that means your **Lotus Miner** isn't ready yet. You need to finish [syncing the chain](https://lotu.sh/en+join-testnet).
## Error: Your computer may not be fast enough
```sh
CAUTION: block production took longer than the block delay. Your computer may not be fast enough to keep up
```
If you see this, that means your computer is too slow and your blocks are not included in the chain, and you will not receive any rewards.
## Error: No space left on device
```sh
lotus-miner sectors pledge
# No space left on device (os error 28)
```
If you see this, that means `pledge-sector` wrote too much data to `$TMPDIR` which by default is the root partition (This is common for Linux setups). Usually your root partition does not get the largest partition of storage so you will need to change the environment variable to something else.
## Error: GPU unused
If you suspect that your GPU is not being used, first make sure it is properly configured as described in the [testing configuration page](hardware-mining.md). Once you've done that (and set the `BELLMAN_CUSTOM_GPU` as appropriate if necessary) you can verify your GPU is being used by running a quick lotus-bench benchmark.
First, to watch GPU utilization run `nvtop` in one terminal, then in a separate terminal, run:
```sh
make bench
./bench sealing --sector-size=2KiB
```
This process uses a fair amount of GPU, and generally takes ~4 minutes to complete. If you do not see any activity in nvtop from lotus during the entire process, it is likely something is misconfigured with your GPU.
## Checking Sync Progress
You can use this command to check how far behind you are on syncing:
```sh
date -d @$(./lotus chain getblock $(./lotus chain head) | jq .Timestamp)
```

149
documentation/en/mining.md
View File

@ -1,149 +0,0 @@
# Storage Mining
Here are instructions to learn how to perform storage mining. For hardware specifications please read [this](https://lotu.sh/en+hardware-mining).
It is useful to [join the Testnet](https://lotu.sh/en+join-testnet) prior to attempting storage mining for the first time.
## Note: Using the Lotus Miner from China
If you are trying to use `lotus-miner` from China. You should set this **environment variable** on your machine.
```sh
export IPFS_GATEWAY="https://proof-parameters.s3.cn-south-1.jdcloud-oss.com/ipfs/"
```
## Get started
Please ensure that at least one **BLS address** (starts with `t3`) in your wallet exists with the following command:
```sh
lotus wallet list
```
If you do not have a bls address, create a new bls wallet:
```sh
lotus wallet new bls
```
With your wallet address:
- Visit the [faucet](http://spacerace.faucet.glif.io/)
- Paste the address you created under REQUEST.
- Press the Request button.
- Run `/lotus-miner init --owner=<blsAddress> --worker=<blsAddress>`
You will have to wait some time for this operation to complete.
## Mining
To mine:
```sh
lotus-miner run
```
If you are downloading **Filecoin Proof Parameters**, the download can take some time.
Get information about your miner:
```sh
lotus-miner info
# example: miner id `t0111`
```
**Seal** random data to start producing **PoSts**:
```sh
lotus-miner sectors pledge
```
- Warning: On Linux configurations, this command will write data to `$TMPDIR` which is not usually the largest partition. You should point the value to a larger partition if possible.
Get **miner power** and **sector usage**:
```sh
lotus state power
# returns total power
lotus state power <miner>
lotus state sectors <miner>
```
## Performance tuning
### `FIL_PROOFS_MAXIMIZE_CACHING=1` Environment variable
This env var can be used with `lotus-miner`, `lotus-worker`, and `lotus-bench` to make the precommit1 step faster at the cost of some memory use (1x sector size)
### `FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1` Environment variable
This env var can be used with `lotus-miner`, `lotus-worker`, and `lotus-bench` to enable experimental precommit2 GPU acceleration
### Setting multiaddresses
Set multiaddresses for the miner to listen on in a miner's `config.toml` file
(by default, it is located at `~/.lotusminer/config.toml`). The `ListenAddresses` in this file should be interface listen addresses (usually `/ip4/0.0.0.0/tcp/PORT`), and the `AnnounceAddresses` should match the addresses being passed to `set-addrs`.
The addresses passed to `set-addrs` parameter in the commands below should be currently active and dialable; confirm they are before entering them.
Once the config file has been updated, set the on-chain record of the miner's listen addresses:
```
lotus-miner actor set-addrs <multiaddr_1> <multiaddr_2> ... <multiaddr_n>
```
This updates the `MinerInfo` object in the miner's actor, which will be looked up
when a client attempts to make a deal. Any number of addresses can be provided.
Example:
```
lotus-miner actor set-addrs /ip4/123.123.73.123/tcp/12345 /ip4/223.223.83.223/tcp/23456
```
# Separate address for windowPoSt messages
WindowPoSt is the mechanism through which storage is verified in Filecoin. It requires miners to submit proofs for all sectors every 24h, which require sending messages to the chain.
Because many other mining related actions require sending messages to the chain, and not all of those are "high value", it may be desirable to use a separate account to send PoSt messages from. This allows for setting lower GasFeeCaps on the lower value messages without creating head-of-line blocking problems for the PoSt messages in congested chain conditions
To set this up, first create a new account, and send it some funds for gas fees:
```sh
lotus wallet new bls
t3defg...
lotus send t3defg... 100
```
Next add the control address
```sh
lotus-miner actor control set t3defg...
Add t3defg...
Pass --really-do-it to actually execute this action
```
Now actually set the addresses
```sh
lotus-miner actor control set --really-do-it t3defg...
Add t3defg...
Message CID: bafy2..
```
Wait for the message to land on chain
```sh
lotus state wait-msg bafy2..
...
Exit Code: 0
...
```
Check miner control address list to make sure the address was correctly setup
```sh
lotus-miner actor control list
name ID key use balance
owner t01111 t3abcd... other 300 FIL
worker t01111 t3abcd... other 300 FIL
control-0 t02222 t3defg... post 100 FIL
```

111
documentation/en/payment-channels.md
View File

@ -1,111 +0,0 @@
# Payment Channels
Payment channels are used to transfer funds between two actors.
For example in lotus a payment channel is created when a client wants to fetch data from a provider.
The client sends vouchers for the payment channel, and the provider sends data in response.
The payment channel is created on-chain with an initial amount.
Vouchers allow the client and the provider to exchange funds incrementally off-chain.
The provider can submit vouchers to chain at any stage.
Either party to the payment channel can settle the payment channel on chain.
After a settlement period (currently 12 hours) either party to the payment channel can call collect on chain.
Collect sends the value of submitted vouchers to the channel recipient (the provider), and refunds the remaining channel balance to the channel creator (the client).
Vouchers have a lane, a nonce and a value, where vouchers with a higher nonce supersede vouchers with a lower nonce in the same lane.
Each deal is created on a different lane.
Note that payment channels and vouchers can be used for any situation in which two parties need to incrementally transfer value between each other off-chain.
## Using the CLI
For example a client creates a payment channel to a provider with value 10 FIL.
```sh
$ lotus paych add-funds <client addr> <provider addr> 10
<channel addr>
```
The client creates a voucher in lane 0 (implied) with nonce 1 (implied) and value 2.
```sh
$ lotus paych voucher create <channel addr> 2
<voucher>
```
The client sends the voucher to the provider and the provider adds the voucher to their local store.
```sh
$ lotus paych voucher add <channel addr> <voucher>
```
The provider sends some data to the client.
The client creates a voucher in lane 0 (implied) with nonce 2 (implied) and value 4.
```sh
$ lotus paych voucher create <channel addr> 4
<voucher>
```
The client sends the voucher to the provider and the provider adds the voucher and sends back more data.
etc.
The client can add value to the channel after it has been created by calling `paych add-funds` with the same client and provider addresses.
```sh
$ lotus paych add-funds <client addr> <provider addr> 5
<channel addr> # Same address as above. Channel now has 15
```
Once the client has received all their data, they may settle the channel.
Note that settlement doesn't have to be done immediately.
For example the client may keep the channel open as long as it wants to continue making deals with the provider.
```sh
$ lotus paych settle <channel addr>
```
The provider can submit vouchers to chain (note that lotus does this automatically when it sees a settle message appear on chain).
The provider may have received many vouchers with incrementally higher values.
The provider should submit the best vouchers. Note that there will be one best voucher for each lane.
```sh
$ lotus paych voucher best-spendable <channel addr>
<voucher>
<voucher>
<voucher>
$ lotus paych voucher submit <channel addr> <voucher>
```
Once the settlement period is over, either the client or provider can call collect to disburse funds.
```sh
$ lotus paych collect <channel addr>
```
Check the status of a channel that is still being created using `lotus paych status-by-from-to`.
```sh
$ lotus paych status-by-from-to <from addr> <to addr>
Creating channel
From: t3sb6xzvs6rhlziatagevxpp3dwapdolurtkpn4kyh3kgoo4tn5o7lutjqlsnvpceztlhxu3lzzfe34rvpsjgq
To: t1zip4sblhyrn4oxygzsm6nafbsynp2avmk3xafea
Pending Amt: 10000
Wait Sentinel: bafy2bzacedk2jidsyxcynusted35t5ipkhu2kpiodtwyjr3pimrhke6f5pqbm
```
Check the status of a channel that has been created using `lotus paych status`.
```sh
$ lotus paych status <channel addr>
Channel exists
Channel: t2nydpzhmeqkmid5smtqnowlr2mr5az6rexpmyv6i
From: t3sb6xzvs6rhlziatagevxpp3dwapdolurtkpn4kyh3kgoo4tn5o7lutjqlsnvpceztlhxu3lzzfe34rvpsjgq
To: t1zip4sblhyrn4oxygzsm6nafbsynp2avmk3xafea
Confirmed Amt: 10000
Pending Amt: 6000
Queued Amt: 3000
Voucher Redeemed Amt: 2000
```

27
documentation/en/retrieving-data.md
View File

@ -1,27 +0,0 @@
# Retrieving Data
> There are recent bug reports with these instructions. If you happen to encounter any problems, please create a [GitHub issue](https://github.com/filecoin-project/lotus/issues/new) and a maintainer will address the problem as soon as they can.
Here are the operations you can perform after you have stored and sealed a **Data CID** with the **Lotus Miner** in the network.
If you would like to learn how to store a **Data CID** on a miner, read the instructions [here](https://lotu.sh/en+storing-data).
## Find by Data CID
```sh
lotus client find <Data CID>
# LOCAL
# RETRIEVAL <miner>@<miner peerId>-<deal funds>-<size>
```
## Retrieve by Data CID
All fields are required.
```sh
lotus client retrieve <Data CID> <outfile>
```
If the outfile does not exist it will be created in the Lotus repository directory.
This command will initiate a **retrieval deal** and write the data to your computer. This process may take 2 to 10 minutes.

54
documentation/en/setting-a-static-port.md
View File

@ -1,54 +0,0 @@
# Static Ports
Depending on how your network is set up, you may need to set a static port to successfully connect to peers to perform storage deals with your **Lotus Miner**.
## Setup
To change the random **swarm port**, you may edit the `config.toml` file located under `$LOTUS_MINER_PATH`. The default location of this file is `$HOME/.lotusminer`.
To change the port to `1347`:
```sh
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/1347", "/ip6/::/tcp/1347"]
```
After changing the port value, restart your **daemon**.
## Announce Addresses
If the **swarm port** is port-forwarded from another address, it is possible to control what addresses
are announced to the network.
```sh
[Libp2p]
AnnounceAddresses = ["/ip4/<public-ip>/tcp/1347"]
```
If non-empty, this array specifies the swarm addresses to announce to the network. If empty, the daemon will announce inferred swarm addresses.
Similarly, it is possible to set `NoAnnounceAddresses` with an array of addresses to not announce to the network.
## Ubuntu's Uncomplicated Firewall
Open firewall manually:
```sh
ufw allow 1347/tcp
```
Or open and modify the profile located at `/etc/ufw/applications.d/lotus-daemon`:
```sh
[Lotus Daemon]
title=Lotus Daemon
description=Lotus Daemon firewall rules
ports=1347/tcp
```
Then run these commands:
```sh
ufw update lotus-daemon
ufw allow lotus-daemon
```

46
documentation/en/setup-troubleshooting.md
View File

@ -1,46 +0,0 @@
# Setup Troubleshooting
## Config: Clearing data
Here is a command that will delete your chain data, stored wallets, stored data and any miners you have set up:
```sh
rm -rf ~/.lotus ~/.lotusminer
```
This command usually resolves any issues with running `lotus` but it is not always required for updates. We will share information about when resetting your chain data and miners is required for an update in the future.
## Error: Failed to connect bootstrap peer
```sh
WARN peermgr peermgr/peermgr.go:131 failed to connect to bootstrap peer: failed to dial : all dials failed
* [/ip4/147.75.80.17/tcp/1347] failed to negotiate security protocol: connected to wrong peer
```
- Try running the build steps again and make sure that you have the latest code from GitHub.
```sh
ERROR hello hello/hello.go:81 other peer has different genesis!
```
- Try deleting your file system's `~/.lotus` directory. Check that it exists with `ls ~/.lotus`.
```sh
- repo is already locked
```
- You already have another lotus daemon running.
## Config: Open files limit
On most systems you can check the open files limit with:
```sh
ulimit -n
```
You can also modify this number by using the `ulimit` command. It gives you the ability to control the resources available for the shell or process started by it. If the number is below 10000, you can change it with the following command prior to starting the Lotus daemon:
```sh
ulimit -n 10000
```

27
documentation/en/storing-data-troubleshooting.md
View File

@ -1,27 +0,0 @@
# Storage Troubleshooting
## Error: Routing: not found
```sh
WARN main lotus/main.go:72 routing: not found
```
- This miner is offline.
## Error: Failed to start deal
```sh
WARN main lotus/main.go:72 failed to start deal: computing commP failed: generating CommP: Piece must be at least 127 bytes
```
- There is a minimum file size of 127 bytes.
## Error: 0kb file response during retrieval
In order to retrieve a file, it must be sealed. Miners can check sealing progress with this command:
```sh
lotus-miner sectors list
```
When sealing is complete, `pSet: NO` will become `pSet: YES`. From now on the **Data CID** is [retrievable](https://lotu.sh/en+retrieving-data) from the **Lotus Miner**.

62
documentation/en/storing-data.md
View File

@ -1,62 +0,0 @@
# Storing Data
> There are recent bug reports with these instructions. If you happen to encounter any problems, please create a [GitHub issue](https://github.com/filecoin-project/lotus/issues/new) and a maintainer will address the problem as soon as they can.
Here are instructions for how to store data on the **Lotus Testnet**.
## Adding a file locally
Adding a file locally allows you to make miner deals on the **Lotus Testnet**.
```sh
lotus client import ./your-example-file.txt
```
Upon success, this command will return a **Data CID**.
## List your local files
The command to see a list of files by `CID`, `name`, `size` in bytes, and `status`:
```sh
lotus client local
```
An example of the output:
```sh
bafkreierupr5ioxn4obwly4i2a5cd2rwxqi6kwmcyyylifxjsmos7hrgpe Development/sample-1.txt 2332 ok
bafkreieuk7h4zs5alzpdyhlph4lxkefowvwdho3a3pml6j7dam5mipzaii Development/sample-2.txt 30618 ok
```
## Make a Miner Deal on Lotus Testnet
Get a list of all miners that can store data:
```sh
lotus state list-miners
```
Get the requirements of a miner you wish to store data with:
```sh
lotus client query-ask <miner>
```
Store a **Data CID** with a miner:
```sh
lotus client deal <Data CID> <miner> <price> <duration>
```
Check the status of a deal:
```sh
lotus client list-deals
```
- The `duration`, which represents how long the miner will keep your file hosted, is represented in blocks. Each block represents 25 seconds.
Upon success, this command will return a **Deal CID**.
The miner will need to **seal** the file before it can be retrieved. If the **Lotus Miner** is not running on a machine designed for sealing, the process will take a very long time.

20
documentation/en/storing-ipfs-integration.md
View File

@ -1,20 +0,0 @@
# IPFS Integration
Lotus supports making deals with data stored in IPFS, without having to re-import it into lotus.
To enable this integration, you need to have an IPFS daemon running in the background.
Then, open up `~/.lotus/config.toml` (or if you manually set `LOTUS_PATH`, look under that directory)
and look for the Client field, and set `UseIpfs` to `true`.
```toml
[Client]
UseIpfs = true
```
After restarting the lotus daemon, you should be able to make deals with data in your IPFS node:
```sh
$ ipfs add -r SomeData
QmSomeData
$ ./lotus client deal QmSomeData t01000 0.0000000001 80000
```

14
documentation/en/updating-lotus.md
View File

@ -1,14 +0,0 @@
# Updating Lotus
If you installed Lotus on your machine, you can upgrade to the latest version by doing the following:
```sh
# get the latest
git pull origin master
# clean and remake the binaries
make clean && make build
# instal binaries in correct location
make install # or sudo make install if necessary
```

2
extern/fil-blst vendored

@ -1 +1 @@
Subproject commit 5f93488fc0dbfb450f2355269f18fc67010d59bb
Subproject commit 8609119cf4595d1741139c24378fcd8bc4f1c475

2
extern/filecoin-ffi vendored

@ -1 +1 @@
Subproject commit f640612a1a1f7a2dd8b3a49e1531db0aa0f63447
Subproject commit 57e38efe4943f09d3127dcf6f0edd614e6acf68e

47
extern/sector-storage/ffiwrapper/sealer_cgo.go vendored
View File

@ -546,34 +546,37 @@ func (sb *Sealer) FinalizeSector(ctx context.Context, sector abi.SectorID, keepU
defer done()
pf, err := openPartialFile(maxPieceSize, paths.Unsealed)
if xerrors.Is(err, os.ErrNotExist) {
return xerrors.Errorf("opening partial file: %w", err)
}
if err == nil {
var at uint64
for sr.HasNext() {
r, err := sr.NextRun()
if err != nil {
_ = pf.Close()
return err
}
var at uint64
for sr.HasNext() {
r, err := sr.NextRun()
if err != nil {
_ = pf.Close()
offset := at
at += r.Len
if !r.Val {
continue
}
err = pf.Free(storiface.PaddedByteIndex(abi.UnpaddedPieceSize(offset).Padded()), abi.UnpaddedPieceSize(r.Len).Padded())
if err != nil {
_ = pf.Close()
return xerrors.Errorf("free partial file range: %w", err)
}
}
if err := pf.Close(); err != nil {
return err
}
offset := at
at += r.Len
if !r.Val {
continue
}
err = pf.Free(storiface.PaddedByteIndex(abi.UnpaddedPieceSize(offset).Padded()), abi.UnpaddedPieceSize(r.Len).Padded())
if err != nil {
_ = pf.Close()
return xerrors.Errorf("free partial file range: %w", err)
} else {
if !xerrors.Is(err, os.ErrNotExist) {
return xerrors.Errorf("opening partial file: %w", err)
}
}
if err := pf.Close(); err != nil {
return err
}
}
paths, done, err := sb.sectors.AcquireSector(ctx, sector, stores.FTCache, 0, stores.PathStorage)

10
extern/sector-storage/localworker.go vendored
View File

@ -26,6 +26,7 @@ var pathTypes = []stores.SectorFileType{stores.FTUnsealed, stores.FTSealed, stor
type WorkerConfig struct {
SealProof abi.RegisteredSealProof
TaskTypes []sealtasks.TaskType
NoSwap bool
}
type LocalWorker struct {
@ -33,6 +34,7 @@ type LocalWorker struct {
storage stores.Store
localStore *stores.Local
sindex stores.SectorIndex
noSwap bool
acceptTasks map[sealtasks.TaskType]struct{}
}
@ -50,6 +52,7 @@ func NewLocalWorker(wcfg WorkerConfig, store stores.Store, local *stores.Local,
storage: store,
localStore: local,
sindex: sindex,
noSwap: wcfg.NoSwap,
acceptTasks: acceptTasks,
}
@ -275,11 +278,16 @@ func (l *LocalWorker) Info(context.Context) (storiface.WorkerInfo, error) {
return storiface.WorkerInfo{}, xerrors.Errorf("getting memory info: %w", err)
}
memSwap := mem.VirtualTotal
if l.noSwap {
memSwap = 0
}
return storiface.WorkerInfo{
Hostname: hostname,
Resources: storiface.WorkerResources{
MemPhysical: mem.Total,
MemSwap: mem.VirtualTotal,
MemSwap: memSwap,
MemReserved: mem.VirtualUsed + mem.Total - mem.Available, // TODO: sub this process
CPUs: uint64(runtime.NumCPU()),
GPUs: gpus,

8
extern/storage-sealing/states_failed.go vendored
View File

@ -38,16 +38,16 @@ func (m *Sealing) checkPreCommitted(ctx statemachine.Context, sector SectorInfo)
tok, _, err := m.api.ChainHead(ctx.Context())
if err != nil {
log.Errorf("handleSealPrecommit1Failed(%d): temp error: %+v", sector.SectorNumber, err)
return nil, true
return nil, false
}
info, err := m.api.StateSectorPreCommitInfo(ctx.Context(), m.maddr, sector.SectorNumber, tok)
if err != nil {
log.Errorf("handleSealPrecommit1Failed(%d): temp error: %+v", sector.SectorNumber, err)
return nil, true
return nil, false
}
return info, false
return info, true
}
func (m *Sealing) handleSealPrecommit1Failed(ctx statemachine.Context, sector SectorInfo) error {
@ -108,7 +108,7 @@ func (m *Sealing) handlePreCommitFailed(ctx statemachine.Context, sector SectorI
}
if pci, is := m.checkPreCommitted(ctx, sector); is && pci != nil {
if sector.PreCommitMessage != nil {
if sector.PreCommitMessage == nil {
log.Warn("sector %d is precommitted on chain, but we don't have precommit message", sector.SectorNumber)
return ctx.Send(SectorPreCommitLanded{TipSet: tok})
}

19
go.mod
View File

@ -11,6 +11,7 @@ require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d
github.com/buger/goterm v0.0.0-20200322175922-2f3e71b85129
github.com/cockroachdb/pebble v0.0.0-20200916222308-4e219a90ba5b
github.com/coreos/go-systemd/v22 v22.0.0
github.com/detailyang/go-fallocate v0.0.0-20180908115635-432fa640bd2e
github.com/dgraph-io/badger/v2 v2.2007.2
@ -21,13 +22,13 @@ require (
github.com/elastic/go-sysinfo v1.3.0
github.com/fatih/color v1.8.0
github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200716204036-cddc56607e1d
github.com/filecoin-project/go-address v0.0.3
github.com/filecoin-project/go-bitfield v0.2.0
github.com/filecoin-project/go-address v0.0.4
github.com/filecoin-project/go-bitfield v0.2.1-0.20200920172649-837cbe6a1ed3
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03
github.com/filecoin-project/go-data-transfer v0.6.5
github.com/filecoin-project/go-data-transfer v0.6.6
github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f
github.com/filecoin-project/go-fil-markets v0.6.2
github.com/filecoin-project/go-fil-markets v0.6.3
github.com/filecoin-project/go-jsonrpc v0.1.2-0.20200822201400-474f4fdccc52
github.com/filecoin-project/go-multistore v0.0.3
github.com/filecoin-project/go-padreader v0.0.0-20200903213702-ed5fae088b20
@ -36,7 +37,7 @@ require (
github.com/filecoin-project/go-statemachine v0.0.0-20200813232949-df9b130df370
github.com/filecoin-project/go-statestore v0.1.0
github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
github.com/filecoin-project/specs-actors v0.9.10
github.com/filecoin-project/specs-actors v0.9.11
github.com/filecoin-project/specs-storage v0.1.1-0.20200907031224-ed2e5cd13796
github.com/filecoin-project/test-vectors/schema v0.0.1
github.com/gbrlsnchs/jwt/v3 v3.0.0-beta.1
@ -54,13 +55,14 @@ require (
github.com/ipfs/go-blockservice v0.1.4-0.20200624145336-a978cec6e834
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.4
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-ds-badger2 v0.1.1-0.20200708190120-187fc06f714e
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-ds-measure v0.1.0
github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459
github.com/ipfs/go-filestore v1.0.0
github.com/ipfs/go-fs-lock v0.0.6
github.com/ipfs/go-graphsync v0.2.0
github.com/ipfs/go-graphsync v0.2.1
github.com/ipfs/go-ipfs-blockstore v1.0.1
github.com/ipfs/go-ipfs-chunker v0.0.5
github.com/ipfs/go-ipfs-ds-help v1.0.0
@ -125,10 +127,13 @@ require (
golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980
golang.org/x/time v0.0.0-20191024005414-555d28b269f0
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
gopkg.in/cheggaaa/pb.v1 v1.0.28
gotest.tools v2.2.0+incompatible
launchpad.net/gocheck v0.0.0-20140225173054-000000000087 // indirect
)
replace github.com/filecoin-project/lotus => ./
replace github.com/golangci/golangci-lint => github.com/golangci/golangci-lint v1.18.0
replace github.com/filecoin-project/filecoin-ffi => ./extern/filecoin-ffi

45
go.sum
View File

@ -112,6 +112,8 @@ github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7
github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/certifi/gocertifi v0.0.0-20200211180108-c7c1fbc02894 h1:JLaf/iINcLyjwbtTsCJjc6rtlASgHeIJPrB6QmwURnA=
github.com/certifi/gocertifi v0.0.0-20200211180108-c7c1fbc02894/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
@ -125,6 +127,14 @@ github.com/clbanning/x2j v0.0.0-20191024224557-825249438eec/go.mod h1:jMjuTZXRI4
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
github.com/cockroachdb/errors v1.2.4 h1:Lap807SXTH5tri2TivECb/4abUkMZC9zRoLarvcKDqs=
github.com/cockroachdb/errors v1.2.4/go.mod h1:rQD95gz6FARkaKkQXUksEje/d9a6wBJoCr5oaCLELYA=
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f h1:o/kfcElHqOiXqcou5a3rIlMc7oJbMQkeLk0VQJ7zgqY=
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f/go.mod h1:i/u985jwjWRlyHXQbwatDASoW0RMlZ/3i9yJHE2xLkI=
github.com/cockroachdb/pebble v0.0.0-20200916222308-4e219a90ba5b h1:OKALTB609+19AM7wsO0k8yMwAqjEIppcnYvyIhA+ZlQ=
github.com/cockroachdb/pebble v0.0.0-20200916222308-4e219a90ba5b/go.mod h1:hU7vhtrqonEphNF+xt8/lHdaBprxmV1h8BOGrd9XwmQ=
github.com/cockroachdb/redact v0.0.0-20200622112456-cd282804bbd3 h1:2+dpIJzYMSbLi0587YXpi8tOJT52qCOI/1I0UNThc/I=
github.com/cockroachdb/redact v0.0.0-20200622112456-cd282804bbd3/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg=
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd h1:qMd81Ts1T2OTKmB4acZcyKaMtRnY5Y44NuXGX2GFJ1w=
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@ -214,20 +224,24 @@ github.com/fatih/color v1.8.0/go.mod h1:3l45GVGkyrnYNl9HoIjnp2NnNWvh6hLAqD8yTfGj
github.com/fd/go-nat v1.0.0/go.mod h1:BTBu/CKvMmOMUPkKVef1pngt2WFH/lg7E6yQnulfp6E=
github.com/filecoin-project/go-address v0.0.3 h1:eVfbdjEbpbzIrbiSa+PiGUY+oDK9HnUn+M1R/ggoHf8=
github.com/filecoin-project/go-address v0.0.3/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
github.com/filecoin-project/go-address v0.0.4 h1:gSNMv0qWwH16fGQs7ycOUrDjY6YCSsgLUl0I0KLjo8w=
github.com/filecoin-project/go-address v0.0.4/go.mod h1:jr8JxKsYx+lQlQZmF5i2U0Z+cGQ59wMIps/8YW/lDj8=
github.com/filecoin-project/go-amt-ipld/v2 v2.1.0 h1:t6qDiuGYYngDqaLc2ZUvdtAg4UNxPeOYaXhBWSNsVaM=
github.com/filecoin-project/go-amt-ipld/v2 v2.1.0/go.mod h1:nfFPoGyX0CU9SkXX8EoCcSuHN1XcbN0c6KBh7yvP5fs=
github.com/filecoin-project/go-bitfield v0.2.0 h1:gCtLcjskIPtdg4NfN7gQZSQF9yrBQ7mkT0qCJxzGI2Q=
github.com/filecoin-project/go-bitfield v0.2.0/go.mod h1:CNl9WG8hgR5mttCnUErjcQjGvuiZjRqK9rHVBsQF4oM=
github.com/filecoin-project/go-bitfield v0.2.1-0.20200920172649-837cbe6a1ed3 h1:HQa4+yCYsLq1TLM0kopeAhSCLbtZ541cWEi5N5rO+9g=
github.com/filecoin-project/go-bitfield v0.2.1-0.20200920172649-837cbe6a1ed3/go.mod h1:CNl9WG8hgR5mttCnUErjcQjGvuiZjRqK9rHVBsQF4oM=
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2 h1:av5fw6wmm58FYMgJeoB/lK9XXrgdugYiTqkdxjTy9k8=
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2/go.mod h1:pqTiPHobNkOVM5thSRsHYjyQfq7O5QSCMhvuu9JoDlg=
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
github.com/filecoin-project/go-data-transfer v0.6.5 h1:oP20la8Z0CLrw0uqvt6xVgw6rOevZeGJ9GNQeC0OCSU=
github.com/filecoin-project/go-data-transfer v0.6.5/go.mod h1:I9Ylb/UiZyqnI41wUoCXq/le0nDLhlwpFQCtNPxEPOA=
github.com/filecoin-project/go-data-transfer v0.6.6 h1:2TccLSxPYJENcYRdov2WvpTvQ1qUMrPkWe8sBrfj36g=
github.com/filecoin-project/go-data-transfer v0.6.6/go.mod h1:C++k1U6+jMQODOaen5OPDo9XQbth9Yq3ie94vNjBJbk=
github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f h1:GxJzR3oRIMTPtpZ0b7QF8FKPK6/iPAc7trhlL5k/g+s=
github.com/filecoin-project/go-fil-commcid v0.0.0-20200716160307-8f644712406f/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
github.com/filecoin-project/go-fil-markets v0.6.2 h1:9Z57KeaQSa1liCmT1pH6SIjrn9mGTDFJXmR2WQVuaiY=
github.com/filecoin-project/go-fil-markets v0.6.2/go.mod h1:wtN4Hc/1hoVCpWhSWYxwUxH3PQtjSkWWuC1nQjiIWog=
github.com/filecoin-project/go-fil-markets v0.6.3 h1:3kTxfquGvk3zQY+hJH1kEA28tRQ47phqSRqOI4+YcQM=
github.com/filecoin-project/go-fil-markets v0.6.3/go.mod h1:Ug1yhGhzTYC6qrpKsR2QpU8QRCeBpwkTA9RICVKuOMM=
github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
@ -254,8 +268,8 @@ github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b
github.com/filecoin-project/go-storedcounter v0.0.0-20200421200003-1c99c62e8a5b/go.mod h1:Q0GQOBtKf1oE10eSXSlhN45kDBdGvEcVOqMiffqX+N8=
github.com/filecoin-project/specs-actors v0.9.4/go.mod h1:BStZQzx5x7TmCkLv0Bpa07U6cPKol6fd3w9KjMPZ6Z4=
github.com/filecoin-project/specs-actors v0.9.7/go.mod h1:wM2z+kwqYgXn5Z7scV1YHLyd1Q1cy0R8HfTIWQ0BFGU=
github.com/filecoin-project/specs-actors v0.9.10 h1:gU0TrRhgkCsBEOP42sGDE7RQuR0Cov9hJhBqq+RJmjU=
github.com/filecoin-project/specs-actors v0.9.10/go.mod h1:czlvLQGEX0fjLLfdNHD7xLymy6L3n7aQzRWzsYGf+ys=
github.com/filecoin-project/specs-actors v0.9.11 h1:TnpG7HAeiUrfj0mJM7UaPW0P2137H62RGof7ftT5Mas=
github.com/filecoin-project/specs-actors v0.9.11/go.mod h1:czlvLQGEX0fjLLfdNHD7xLymy6L3n7aQzRWzsYGf+ys=
github.com/filecoin-project/specs-storage v0.1.1-0.20200907031224-ed2e5cd13796 h1:dJsTPWpG2pcTeojO2pyn0c6l+x/3MZYCBgo/9d11JEk=
github.com/filecoin-project/specs-storage v0.1.1-0.20200907031224-ed2e5cd13796/go.mod h1:nJRRM7Aa9XVvygr3W9k6xGF46RWzr2zxF/iGoAIfA/g=
github.com/filecoin-project/test-vectors/schema v0.0.1 h1:5fNF76nl4qolEvcIsjc0kUADlTMVHO73tW4kXXPnsus=
@ -272,12 +286,16 @@ github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWo
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/gbrlsnchs/jwt/v3 v3.0.0-beta.1 h1:EzDjxMg43q1tA2c0MV3tNbaontnHLplHyFF6M5KiVP0=
github.com/gbrlsnchs/jwt/v3 v3.0.0-beta.1/go.mod h1:0eHX/BVySxPc6SE2mZRoppGq7qcEagxdmQnA3dzork8=
github.com/getsentry/raven-go v0.2.0 h1:no+xWJRb5ZI7eE8TWgIq1jLulQiIoLG0IfYxv5JYMGs=
github.com/getsentry/raven-go v0.2.0/go.mod h1:KungGk8q33+aIAZUIVWZDr2OfAEBsO49PX4NzFV5kcQ=
github.com/ghemawat/stream v0.0.0-20171120220530-696b145b53b9/go.mod h1:106OIgooyS7OzLDOpUGgm9fA3bQENb/cFSyyBmMoJDs=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.10.0 h1:dXFJfIHVvUcpSgDOV+Ne6t7jXri8Tfv2uOLHUZ2XNuo=
@ -339,6 +357,8 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.2-0.20190904063534-ff6b7dc882cf h1:gFVkHXmVAhEbxZVDln5V9GKrLaluNoFHDbrZwAWZgws=
github.com/golang/snappy v0.0.2-0.20190904063534-ff6b7dc882cf/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
@ -479,6 +499,8 @@ github.com/ipfs/go-datastore v0.4.1/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13X
github.com/ipfs/go-datastore v0.4.2/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA=
github.com/ipfs/go-datastore v0.4.4 h1:rjvQ9+muFaJ+QZ7dN5B1MSDNQ0JVZKkkES/rMZmA8X8=
github.com/ipfs/go-datastore v0.4.4/go.mod h1:SX/xMIKoCszPqp+z9JhPYCmoOoXTvaa13XEbGtsFUhA=
github.com/ipfs/go-datastore v0.4.5 h1:cwOUcGMLdLPWgu3SlrCckCMznaGADbPqE0r8h768/Dg=
github.com/ipfs/go-datastore v0.4.5/go.mod h1:eXTcaaiN6uOlVCLS9GjJUJtlvJfM3xk23w3fyfrmmJs=
github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk=
github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps=
github.com/ipfs/go-ds-badger v0.0.2/go.mod h1:Y3QpeSFWQf6MopLTiZD+VT6IC1yZqaGmjvRcKeSGij8=
@ -497,13 +519,16 @@ github.com/ipfs/go-ds-leveldb v0.4.2 h1:QmQoAJ9WkPMUfBLnu1sBVy0xWWlJPg0m4kRAiJL9
github.com/ipfs/go-ds-leveldb v0.4.2/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s=
github.com/ipfs/go-ds-measure v0.1.0 h1:vE4TyY4aeLeVgnnPBC5QzKIjKrqzha0NCujTfgvVbVQ=
github.com/ipfs/go-ds-measure v0.1.0/go.mod h1:1nDiFrhLlwArTME1Ees2XaBOl49OoCgd2A3f8EchMSY=
github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459 h1:W3YMLEvOXqdW+sYMiguhWP6txJwQvIQqhvpU8yAMGQs=
github.com/ipfs/go-ds-pebble v0.0.2-0.20200921225637-ce220f8ac459/go.mod h1:oh4liWHulKcDKVhCska5NLelE3MatWl+1FwSz3tY91g=
github.com/ipfs/go-filestore v1.0.0 h1:QR7ekKH+q2AGiWDc7W2Q0qHuYSRZGUJqUn0GsegEPb0=
github.com/ipfs/go-filestore v1.0.0/go.mod h1:/XOCuNtIe2f1YPbiXdYvD0BKLA0JR1MgPiFOdcuu9SM=
github.com/ipfs/go-fs-lock v0.0.6 h1:sn3TWwNVQqSeNjlWy6zQ1uUGAZrV3hPOyEA6y1/N2a0=
github.com/ipfs/go-fs-lock v0.0.6/go.mod h1:OTR+Rj9sHiRubJh3dRhD15Juhd/+w6VPOY28L7zESmM=
github.com/ipfs/go-graphsync v0.1.0/go.mod h1:jMXfqIEDFukLPZHqDPp8tJMbHO9Rmeb9CEGevngQbmE=
github.com/ipfs/go-graphsync v0.2.0 h1:x94MvHLNuRwBlZzVal7tR1RYK7T7H6bqQLPopxDbIF0=
github.com/ipfs/go-graphsync v0.2.0/go.mod h1:gEBvJUNelzMkaRPJTpg/jaKN4AQW/7wDWu0K92D8o10=
github.com/ipfs/go-graphsync v0.2.1 h1:MdehhqBSuTI2LARfKLkpYnt0mUrqHs/mtuDnESXHBfU=
github.com/ipfs/go-graphsync v0.2.1/go.mod h1:gEBvJUNelzMkaRPJTpg/jaKN4AQW/7wDWu0K92D8o10=
github.com/ipfs/go-hamt-ipld v0.1.1 h1:0IQdvwnAAUKmDE+PMJa5y1QiwOPHpI9+eAbQEEEYthk=
github.com/ipfs/go-hamt-ipld v0.1.1/go.mod h1:1EZCr2v0jlCnhpa+aZ0JZYp8Tt2w16+JJOAVz17YcDk=
github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=
@ -1493,6 +1518,8 @@ golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd h1:zkO/Lhoka23X63N9OSzpSeROEUQ5ODw47tM3YWjygbs=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200513190911-00229845015e h1:rMqLP+9XLy+LdbCXHjJHAmTfXCr93W7oruWA6Hq1Alc=
golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@ -1511,6 +1538,7 @@ golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCc
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
@ -1612,6 +1640,7 @@ golang.org/x/sys v0.0.0-20190902133755-9109b7679e13/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191025021431-6c3a3bfe00ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=

4
miner/miner.go
View File

@ -211,6 +211,8 @@ minerLoop:
base = prebase
}
base.NullRounds += injectNulls // testing
if base.TipSet.Equals(lastBase.TipSet) && lastBase.NullRounds == base.NullRounds {
log.Warnf("BestMiningCandidate from the previous round: %s (nulls:%d)", lastBase.TipSet.Cids(), lastBase.NullRounds)
if !m.niceSleep(time.Duration(build.BlockDelaySecs) * time.Second) {
@ -219,8 +221,6 @@ minerLoop:
continue
}
base.NullRounds += injectNulls // testing
b, err := m.mineOne(ctx, base)
if err != nil {
log.Errorf("mining block failed: %+v", err)

2
node/builder.go
View File

@ -35,6 +35,7 @@ import (
"github.com/filecoin-project/lotus/chain/gen/slashfilter"
"github.com/filecoin-project/lotus/chain/market"
"github.com/filecoin-project/lotus/chain/messagepool"
"github.com/filecoin-project/lotus/chain/messagesigner"
"github.com/filecoin-project/lotus/chain/metrics"
"github.com/filecoin-project/lotus/chain/stmgr"
"github.com/filecoin-project/lotus/chain/store"
@ -259,6 +260,7 @@ func Online() Option {
Override(new(*store.ChainStore), modules.ChainStore),
Override(new(*stmgr.StateManager), stmgr.NewStateManager),
Override(new(*wallet.Wallet), wallet.NewWallet),
Override(new(*messagesigner.MessageSigner), messagesigner.NewMessageSigner),
Override(new(dtypes.ChainGCLocker), blockstore.NewGCLocker),
Override(new(dtypes.ChainGCBlockstore), modules.ChainGCBlockstore),

65
node/impl/full/mpool.go
View File

@ -10,8 +10,7 @@ import (
"golang.org/x/xerrors"
"github.com/filecoin-project/lotus/api"
"github.com/filecoin-project/lotus/chain/messagepool"
"github.com/filecoin-project/lotus/chain/store"
"github.com/filecoin-project/lotus/chain/messagesigner"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/node/modules/dtypes"
)
@ -22,9 +21,7 @@ type MpoolAPI struct {
WalletAPI
GasAPI
Chain *store.ChainStore
Mpool *messagepool.MessagePool
MessageSigner *messagesigner.MessageSigner
PushLocks *dtypes.MpoolLocker
}
@ -114,12 +111,14 @@ func (a *MpoolAPI) MpoolPush(ctx context.Context, smsg *types.SignedMessage) (ci
}
func (a *MpoolAPI) MpoolPushMessage(ctx context.Context, msg *types.Message, spec *api.MessageSendSpec) (*types.SignedMessage, error) {
cp := *msg
msg = &cp
inMsg := *msg
fromA, err := a.Stmgr.ResolveToKeyAddress(ctx, msg.From, nil)
if err != nil {
return nil, xerrors.Errorf("getting key address: %w", err)
}
{
fromA, err := a.Stmgr.ResolveToKeyAddress(ctx, msg.From, nil)
if err != nil {
return nil, xerrors.Errorf("getting key address: %w", err)
}
done, err := a.PushLocks.TakeLock(ctx, fromA)
if err != nil {
return nil, xerrors.Errorf("taking lock: %w", err)
@ -131,7 +130,7 @@ func (a *MpoolAPI) MpoolPushMessage(ctx context.Context, msg *types.Message, spe
return nil, xerrors.Errorf("MpoolPushMessage expects message nonce to be 0, was %d", msg.Nonce)
}
msg, err := a.GasAPI.GasEstimateMessageGas(ctx, msg, spec, types.EmptyTSK)
msg, err = a.GasAPI.GasEstimateMessageGas(ctx, msg, spec, types.EmptyTSK)
if err != nil {
return nil, xerrors.Errorf("GasEstimateMessageGas error: %w", err)
}
@ -143,33 +142,31 @@ func (a *MpoolAPI) MpoolPushMessage(ctx context.Context, msg *types.Message, spe
inJson, outJson)
}
sign := func(from address.Address, nonce uint64) (*types.SignedMessage, error) {
msg.Nonce = nonce
if msg.From.Protocol() == address.ID {
log.Warnf("Push from ID address (%s), adjusting to %s", msg.From, from)
msg.From = from
}
b, err := a.WalletBalance(ctx, msg.From)
if err != nil {
return nil, xerrors.Errorf("mpool push: getting origin balance: %w", err)
}
if b.LessThan(msg.Value) {
return nil, xerrors.Errorf("mpool push: not enough funds: %s < %s", b, msg.Value)
}
return a.WalletSignMessage(ctx, from, msg)
if msg.From.Protocol() == address.ID {
log.Warnf("Push from ID address (%s), adjusting to %s", msg.From, fromA)
msg.From = fromA
}
var m *types.SignedMessage
again:
m, err = a.Mpool.PushWithNonce(ctx, msg.From, sign)
if err == messagepool.ErrTryAgain {
log.Debugf("temporary failure while pushing message: %s; retrying", err)
goto again
b, err := a.WalletBalance(ctx, msg.From)
if err != nil {
return nil, xerrors.Errorf("mpool push: getting origin balance: %w", err)
}
return m, err
if b.LessThan(msg.Value) {
return nil, xerrors.Errorf("mpool push: not enough funds: %s < %s", b, msg.Value)
}
smsg, err := a.MessageSigner.SignMessage(ctx, msg)
if err != nil {
return nil, xerrors.Errorf("mpool push: failed to sign message: %w", err)
}
_, err = a.Mpool.Push(smsg)
if err != nil {
return nil, xerrors.Errorf("mpool push: failed to push message: %w", err)
}
return smsg, err
}
func (a *MpoolAPI) MpoolGetNonce(ctx context.Context, addr address.Address) (uint64, error) {

19
node/impl/full/sync.go
View File

@ -126,3 +126,22 @@ func (a *SyncAPI) SyncCheckBad(ctx context.Context, bcid cid.Cid) (string, error
return reason, nil
}
func (a *SyncAPI) SyncValidateTipset(ctx context.Context, tsk types.TipSetKey) (bool, error) {
ts, err := a.Syncer.ChainStore().LoadTipSet(tsk)
if err != nil {
return false, err
}
fts, err := a.Syncer.ChainStore().TryFillTipSet(ts)
if err != nil {
return false, err
}
err = a.Syncer.ValidateTipSet(ctx, fts, false)
if err != nil {
return false, err
}
return true, nil
}

4
node/impl/full/wallet.go
View File

@ -90,3 +90,7 @@ func (a *WalletAPI) WalletImport(ctx context.Context, ki *types.KeyInfo) (addres
func (a *WalletAPI) WalletDelete(ctx context.Context, addr address.Address) error {
return a.Wallet.DeleteKey(addr)
}
func (a *WalletAPI) WalletValidateAddress(ctx context.Context, str string) (address.Address, error) {
return address.NewFromString(str)
}

5
node/modules/chain.go
View File

@ -18,6 +18,7 @@ import (
"github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/lotus/chain"
"github.com/filecoin-project/lotus/chain/beacon"
"github.com/filecoin-project/lotus/chain/exchange"
@ -157,6 +158,10 @@ func SetGenesis(cs *store.ChainStore, g Genesis) (dtypes.AfterGenesisSet, error)
}
func NetworkName(mctx helpers.MetricsCtx, lc fx.Lifecycle, cs *store.ChainStore, _ dtypes.AfterGenesisSet) (dtypes.NetworkName, error) {
if !build.Devnet {
return "testnetnet", nil
}
ctx := helpers.LifecycleCtx(mctx, lc)
netName, err := stmgr.GetNetworkName(ctx, stmgr.NewStateManager(cs), cs.GetHeaviestTipSet().ParentState())

3
node/modules/storageminer.go
View File

@ -109,6 +109,9 @@ func MinerID(ma dtypes.MinerAddress) (dtypes.MinerID, error) {
}
func StorageNetworkName(ctx helpers.MetricsCtx, a lapi.FullNode) (dtypes.NetworkName, error) {
if !build.Devnet {
return "testnetnet", nil
}
return a.StateNetworkName(ctx)
}

34
node/repo/fsrepo.go
View File

@ -65,7 +65,8 @@ var ErrRepoExists = xerrors.New("repo exists")
// FsRepo is struct for repo, use NewFS to create
type FsRepo struct {
path string
path string
configPath string
}
var _ Repo = &FsRepo{}
@ -78,10 +79,15 @@ func NewFS(path string) (*FsRepo, error) {
}
return &FsRepo{
path: path,
path: path,
configPath: filepath.Join(path, fsConfig),
}, nil
}
func (fsr *FsRepo) SetConfigPath(cfgPath string) {
fsr.configPath = cfgPath
}
func (fsr *FsRepo) Exists() (bool, error) {
_, err := os.Stat(filepath.Join(fsr.path, fsDatastore))
notexist := os.IsNotExist(err)
@ -115,9 +121,7 @@ func (fsr *FsRepo) Init(t RepoType) error {
}
func (fsr *FsRepo) initConfig(t RepoType) error {
cfgP := filepath.Join(fsr.path, fsConfig)
_, err := os.Stat(cfgP)
_, err := os.Stat(fsr.configPath)
if err == nil {
// exists
return nil
@ -125,7 +129,7 @@ func (fsr *FsRepo) initConfig(t RepoType) error {
return err
}
c, err := os.Create(cfgP)
c, err := os.Create(fsr.configPath)
if err != nil {
return err
}
@ -215,16 +219,18 @@ func (fsr *FsRepo) Lock(repoType RepoType) (LockedRepo, error) {
return nil, xerrors.Errorf("could not lock the repo: %w", err)
}
return &fsLockedRepo{
path: fsr.path,
repoType: repoType,
closer: closer,
path: fsr.path,
configPath: fsr.configPath,
repoType: repoType,
closer: closer,
}, nil
}
type fsLockedRepo struct {
path string
repoType RepoType
closer io.Closer
path string
configPath string
repoType RepoType
closer io.Closer
ds map[string]datastore.Batching
dsErr error
@ -277,7 +283,7 @@ func (fsr *fsLockedRepo) Config() (interface{}, error) {
}
func (fsr *fsLockedRepo) loadConfigFromDisk() (interface{}, error) {
return config.FromFile(fsr.join(fsConfig), defConfForType(fsr.repoType))
return config.FromFile(fsr.configPath, defConfForType(fsr.repoType))
}
func (fsr *fsLockedRepo) SetConfig(c func(interface{})) error {
@ -306,7 +312,7 @@ func (fsr *fsLockedRepo) SetConfig(c func(interface{})) error {
}
// write buffer of TOML bytes to config file
err = ioutil.WriteFile(fsr.join(fsConfig), buf.Bytes(), 0644)
err = ioutil.WriteFile(fsr.configPath, buf.Bytes(), 0644)
if err != nil {
return err
}

537
storage/wdpost_changehandler.go Normal file
View File

@ -0,0 +1,537 @@
package storage
import (
"context"
"sync"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/lotus/chain/actors/builtin/miner"
"github.com/filecoin-project/go-state-types/dline"
"github.com/filecoin-project/lotus/chain/types"
)
const SubmitConfidence = 4
type CompleteGeneratePoSTCb func(posts []miner.SubmitWindowedPoStParams, err error)
type CompleteSubmitPoSTCb func(err error)
type changeHandlerAPI interface {
StateMinerProvingDeadline(context.Context, address.Address, types.TipSetKey) (*dline.Info, error)
startGeneratePoST(ctx context.Context, ts *types.TipSet, deadline *dline.Info, onComplete CompleteGeneratePoSTCb) context.CancelFunc
startSubmitPoST(ctx context.Context, ts *types.TipSet, deadline *dline.Info, posts []miner.SubmitWindowedPoStParams, onComplete CompleteSubmitPoSTCb) context.CancelFunc
onAbort(ts *types.TipSet, deadline *dline.Info)
failPost(err error, ts *types.TipSet, deadline *dline.Info)
}
type changeHandler struct {
api changeHandlerAPI
actor address.Address
proveHdlr *proveHandler
submitHdlr *submitHandler
}
func newChangeHandler(api changeHandlerAPI, actor address.Address) *changeHandler {
posts := newPostsCache()
p := newProver(api, posts)
s := newSubmitter(api, posts)
return &changeHandler{api: api, actor: actor, proveHdlr: p, submitHdlr: s}
}
func (ch *changeHandler) start() {
go ch.proveHdlr.run()
go ch.submitHdlr.run()
}
func (ch *changeHandler) update(ctx context.Context, revert *types.TipSet, advance *types.TipSet) error {
// Get the current deadline period
di, err := ch.api.StateMinerProvingDeadline(ctx, ch.actor, advance.Key())
if err != nil {
return err
}
if !di.PeriodStarted() {
return nil // not proving anything yet
}
hc := &headChange{
ctx: ctx,
revert: revert,
advance: advance,
di: di,
}
select {
case ch.proveHdlr.hcs <- hc:
case <-ch.proveHdlr.shutdownCtx.Done():
case <-ctx.Done():
}
select {
case ch.submitHdlr.hcs <- hc:
case <-ch.submitHdlr.shutdownCtx.Done():
case <-ctx.Done():
}
return nil
}
func (ch *changeHandler) shutdown() {
ch.proveHdlr.shutdown()
ch.submitHdlr.shutdown()
}
func (ch *changeHandler) currentTSDI() (*types.TipSet, *dline.Info) {
return ch.submitHdlr.currentTSDI()
}
// postsCache keeps a cache of PoSTs for each proving window
type postsCache struct {
added chan *postInfo
lk sync.RWMutex
cache map[abi.ChainEpoch][]miner.SubmitWindowedPoStParams
}
func newPostsCache() *postsCache {
return &postsCache{
added: make(chan *postInfo, 16),
cache: make(map[abi.ChainEpoch][]miner.SubmitWindowedPoStParams),
}
}
func (c *postsCache) add(di *dline.Info, posts []miner.SubmitWindowedPoStParams) {
c.lk.Lock()
defer c.lk.Unlock()
// TODO: clear cache entries older than chain finality
c.cache[di.Open] = posts
c.added <- &postInfo{
di: di,
posts: posts,
}
}
func (c *postsCache) get(di *dline.Info) ([]miner.SubmitWindowedPoStParams, bool) {
c.lk.RLock()
defer c.lk.RUnlock()
posts, ok := c.cache[di.Open]
return posts, ok
}
type headChange struct {
ctx context.Context
revert *types.TipSet
advance *types.TipSet
di *dline.Info
}
type currentPost struct {
di *dline.Info
abort context.CancelFunc
}
type postResult struct {
ts *types.TipSet
currPost *currentPost
posts []miner.SubmitWindowedPoStParams
err error
}
// proveHandler generates proofs
type proveHandler struct {
api changeHandlerAPI
posts *postsCache
postResults chan *postResult
hcs chan *headChange
current *currentPost
shutdownCtx context.Context
shutdown context.CancelFunc
// Used for testing
processedHeadChanges chan *headChange
processedPostResults chan *postResult
}
func newProver(
api changeHandlerAPI,
posts *postsCache,
) *proveHandler {
ctx, cancel := context.WithCancel(context.Background())
return &proveHandler{
api: api,
posts: posts,
postResults: make(chan *postResult),
hcs: make(chan *headChange),
shutdownCtx: ctx,
shutdown: cancel,
}
}
func (p *proveHandler) run() {
// Abort proving on shutdown
defer func() {
if p.current != nil {
p.current.abort()
}
}()
for p.shutdownCtx.Err() == nil {
select {
case <-p.shutdownCtx.Done():
return
case hc := <-p.hcs:
// Head changed
p.processHeadChange(hc.ctx, hc.advance, hc.di)
if p.processedHeadChanges != nil {
p.processedHeadChanges <- hc
}
case res := <-p.postResults:
// Proof generation complete
p.processPostResult(res)
if p.processedPostResults != nil {
p.processedPostResults <- res
}
}
}
}
func (p *proveHandler) processHeadChange(ctx context.Context, newTS *types.TipSet, di *dline.Info) {
// If the post window has expired, abort the current proof
if p.current != nil && newTS.Height() >= p.current.di.Close {
// Cancel the context on the current proof
p.current.abort()
// Clear out the reference to the proof so that we can immediately
// start generating a new proof, without having to worry about state
// getting clobbered when the abort completes
p.current = nil
}
// Only generate one proof at a time
if p.current != nil {
return
}
// If the proof for the current post window has been generated, check the
// next post window
_, complete := p.posts.get(di)
for complete {
di = nextDeadline(di)
_, complete = p.posts.get(di)
}
// Check if the chain is above the Challenge height for the post window
if newTS.Height() < di.Challenge {
return
}
p.current = &currentPost{di: di}
curr := p.current
p.current.abort = p.api.startGeneratePoST(ctx, newTS, di, func(posts []miner.SubmitWindowedPoStParams, err error) {
p.postResults <- &postResult{ts: newTS, currPost: curr, posts: posts, err: err}
})
}
func (p *proveHandler) processPostResult(res *postResult) {
di := res.currPost.di
if res.err != nil {
// Proving failed so inform the API
p.api.failPost(res.err, res.ts, di)
log.Warnf("Aborted window post Proving (Deadline: %+v)", di)
p.api.onAbort(res.ts, di)
// Check if the current post has already been aborted
if p.current == res.currPost {
// If the current post was not already aborted, setting it to nil
// marks it as complete so that a new post can be started
p.current = nil
}
return
}
// Completed processing this proving window
p.current = nil
// Add the proofs to the cache
p.posts.add(di, res.posts)
}
type submitResult struct {
pw *postWindow
err error
}
type SubmitState string
const (
SubmitStateStart SubmitState = "SubmitStateStart"
SubmitStateSubmitting SubmitState = "SubmitStateSubmitting"
SubmitStateComplete SubmitState = "SubmitStateComplete"
)
type postWindow struct {
ts *types.TipSet
di *dline.Info
submitState SubmitState
abort context.CancelFunc
}
type postInfo struct {
di *dline.Info
posts []miner.SubmitWindowedPoStParams
}
// submitHandler submits proofs on-chain
type submitHandler struct {
api changeHandlerAPI
posts *postsCache
submitResults chan *submitResult
hcs chan *headChange
postWindows map[abi.ChainEpoch]*postWindow
getPostWindowReqs chan *getPWReq
shutdownCtx context.Context
shutdown context.CancelFunc
currentCtx context.Context
currentTS *types.TipSet
currentDI *dline.Info
getTSDIReq chan chan *tsdi
// Used for testing
processedHeadChanges chan *headChange
processedSubmitResults chan *submitResult
processedPostReady chan *postInfo
}
func newSubmitter(
api changeHandlerAPI,
posts *postsCache,
) *submitHandler {
ctx, cancel := context.WithCancel(context.Background())
return &submitHandler{
api: api,
posts: posts,
submitResults: make(chan *submitResult),
hcs: make(chan *headChange),
postWindows: make(map[abi.ChainEpoch]*postWindow),
getPostWindowReqs: make(chan *getPWReq),
getTSDIReq: make(chan chan *tsdi),
shutdownCtx: ctx,
shutdown: cancel,
}
}
func (s *submitHandler) run() {
// On shutdown, abort in-progress submits
defer func() {
for _, pw := range s.postWindows {
if pw.abort != nil {
pw.abort()
}
}
}()
for s.shutdownCtx.Err() == nil {
select {
case <-s.shutdownCtx.Done():
return
case hc := <-s.hcs:
// Head change
s.processHeadChange(hc.ctx, hc.revert, hc.advance, hc.di)
if s.processedHeadChanges != nil {
s.processedHeadChanges <- hc
}
case pi := <-s.posts.added:
// Proof generated
s.processPostReady(pi)
if s.processedPostReady != nil {
s.processedPostReady <- pi
}
case res := <-s.submitResults:
// Submit complete
s.processSubmitResult(res)
if s.processedSubmitResults != nil {
s.processedSubmitResults <- res
}
case pwreq := <-s.getPostWindowReqs:
// used by getPostWindow() to sync with run loop
pwreq.out <- s.postWindows[pwreq.di.Open]
case out := <-s.getTSDIReq:
// used by currentTSDI() to sync with run loop
out <- &tsdi{ts: s.currentTS, di: s.currentDI}
}
}
}
// processHeadChange is called when the chain head changes
func (s *submitHandler) processHeadChange(ctx context.Context, revert *types.TipSet, advance *types.TipSet, di *dline.Info) {
s.currentCtx = ctx
s.currentTS = advance
s.currentDI = di
// Start tracking the current post window if we're not already
// TODO: clear post windows older than chain finality
if _, ok := s.postWindows[di.Open]; !ok {
s.postWindows[di.Open] = &postWindow{
di: di,
ts: advance,
submitState: SubmitStateStart,
}
}
// Apply the change to all post windows
for _, pw := range s.postWindows {
s.processHeadChangeForPW(ctx, revert, advance, pw)
}
}
func (s *submitHandler) processHeadChangeForPW(ctx context.Context, revert *types.TipSet, advance *types.TipSet, pw *postWindow) {
revertedToPrevDL := revert != nil && revert.Height() < pw.di.Open
expired := advance.Height() >= pw.di.Close
// If the chain was reverted back to the previous deadline, or if the post
// window has expired, abort submit
if pw.submitState == SubmitStateSubmitting && (revertedToPrevDL || expired) {
// Replace the aborted postWindow with a new one so that we can
// submit again at any time without the state getting clobbered
// when the abort completes
abort := pw.abort
if abort != nil {
pw = &postWindow{
di: pw.di,
ts: advance,
submitState: SubmitStateStart,
}
s.postWindows[pw.di.Open] = pw
// Abort the current submit
abort()
}
} else if pw.submitState == SubmitStateComplete && revertedToPrevDL {
// If submit for this deadline has completed, but the chain was
// reverted back to the previous deadline, reset the submit state to the
// starting state, so that it can be resubmitted
pw.submitState = SubmitStateStart
}
// Submit the proof to chain if the proof has been generated and the chain
// height is above confidence
s.submitIfReady(ctx, advance, pw)
}
// processPostReady is called when a proof generation completes
func (s *submitHandler) processPostReady(pi *postInfo) {
pw, ok := s.postWindows[pi.di.Open]
if ok {
s.submitIfReady(s.currentCtx, s.currentTS, pw)
}
}
// submitIfReady submits a proof if the chain is high enough and the proof
// has been generated for this deadline
func (s *submitHandler) submitIfReady(ctx context.Context, advance *types.TipSet, pw *postWindow) {
// If the window has expired, there's nothing more to do.
if advance.Height() >= pw.di.Close {
return
}
// Check if we're already submitting, or already completed submit
if pw.submitState != SubmitStateStart {
return
}
// Check if we've reached the confidence height to submit
if advance.Height() < pw.di.Open+SubmitConfidence {
return
}
// Check if the proofs have been generated for this deadline
posts, ok := s.posts.get(pw.di)
if !ok {
return
}
// If there was nothing to prove, move straight to the complete state
if len(posts) == 0 {
pw.submitState = SubmitStateComplete
return
}
// Start submitting post
pw.submitState = SubmitStateSubmitting
pw.abort = s.api.startSubmitPoST(ctx, advance, pw.di, posts, func(err error) {
s.submitResults <- &submitResult{pw: pw, err: err}
})
}
// processSubmitResult is called with the response to a submit
func (s *submitHandler) processSubmitResult(res *submitResult) {
if res.err != nil {
// Submit failed so inform the API and go back to the start state
s.api.failPost(res.err, res.pw.ts, res.pw.di)
log.Warnf("Aborted window post Submitting (Deadline: %+v)", res.pw.di)
s.api.onAbort(res.pw.ts, res.pw.di)
res.pw.submitState = SubmitStateStart
return
}
// Submit succeeded so move to complete state
res.pw.submitState = SubmitStateComplete
}
type tsdi struct {
ts *types.TipSet
di *dline.Info
}
func (s *submitHandler) currentTSDI() (*types.TipSet, *dline.Info) {
out := make(chan *tsdi)
s.getTSDIReq <- out
res := <-out
return res.ts, res.di
}
type getPWReq struct {
di *dline.Info
out chan *postWindow
}
func (s *submitHandler) getPostWindow(di *dline.Info) *postWindow {
out := make(chan *postWindow)
s.getPostWindowReqs <- &getPWReq{di: di, out: out}
return <-out
}
// nextDeadline gets deadline info for the subsequent deadline
func nextDeadline(currentDeadline *dline.Info) *dline.Info {
periodStart := currentDeadline.PeriodStart
newDeadline := currentDeadline.Index + 1
if newDeadline == miner.WPoStPeriodDeadlines {
newDeadline = 0
periodStart = periodStart + miner.WPoStProvingPeriod
}
return NewDeadlineInfo(periodStart, newDeadline, currentDeadline.CurrentEpoch)
}
func NewDeadlineInfo(periodStart abi.ChainEpoch, deadlineIdx uint64, currEpoch abi.ChainEpoch) *dline.Info {
return dline.NewInfo(periodStart, deadlineIdx, currEpoch, miner.WPoStPeriodDeadlines, miner.WPoStProvingPeriod, miner.WPoStChallengeWindow, miner.WPoStChallengeLookback, miner.FaultDeclarationCutoff)
}

1173
storage/wdpost_changehandler_test.go Normal file
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More