Merge pull request #1 from vulcanize/ian/get_many_blockstore

GetMany badgerdb blockstore method

.gitignore

@@ -52,3 +52,4 @@ dist/
 # The following files are checked into git and result
 # in dirty git state if removed from the docker context
 !extern/filecoin-ffi/rust/filecoin.pc
+!extern/test-vectors

CHANGELOG.md

@@ -1,135 +1,10 @@
 # Lotus changelog
 
-# v1.23.1-rc4 / 2023-06-05
-
-This is the 4th release candidate of the upcoming optional feature release of Lotus v1.23.1.
-
-**☢️ Upgrade Warnings ☢️**
-
-- If you are upgrading to this release candidate from Lotus v1.22.1, please make sure to read the upgrade warnings section in the [v1.23.0 release first.](https://github.com/filecoin-project/lotus/releases/tag/v1.23.0)
+# UNRELEASED
 
 ## New features
-
-- feat: daemon: Auto-resume interrupted snapshot imports ([filecoin-project/lotus#10636](https://github.com/filecoin-project/lotus/pull/10636))
-- feat: VM Execution Lanes ([filecoin-project/lotus#10551](https://github.com/filecoin-project/lotus/pull/10551))
-- feat: chainstore: batch writes of tipsets ([filecoin-project/lotus#10800](https://github.com/filecoin-project/lotus/pull/10800))
-- Add API and CLI to unseal sector (#10626) ([filecoin-project/lotus#10626](https://github.com/filecoin-project/lotus/pull/10626))
-- feat: sealing: Split PCA/PCB batches if gas used exceeds block limit ([filecoin-project/lotus#10647](https://github.com/filecoin-project/lotus/pull/10647))
-- Add feature to stagger sector prove commit submission (#10543) ([filecoin-project/lotus#10543](https://github.com/filecoin-project/lotus/pull/10543))
-- Set default for MaxSectorProveCommitsSubmittedPerEpoch ([filecoin-project/lotus#10728](https://github.com/filecoin-project/lotus/pull/10728))
-- fix: storage: Remove temp fetching files after failed fetch ([filecoin-project/lotus#10661](https://github.com/filecoin-project/lotus/pull/10661))
-- feat: worker: Ensure tempdir exists (#10433) ([filecoin-project/lotus#10433](https://github.com/filecoin-project/lotus/pull/10433))
-- feat: sync: harden chain sync (#10756) ([filecoin-project/lotus#10756](https://github.com/filecoin-project/lotus/pull/10756))
-- feat: populate the index on snapshot import ([filecoin-project/lotus#10556](https://github.com/filecoin-project/lotus/pull/10556))
-- feat:chain: Message Index ([filecoin-project/lotus#10452](https://github.com/filecoin-project/lotus/pull/10452))
-- feat: Add small cache to execution traces ([filecoin-project/lotus#10517](https://github.com/filecoin-project/lotus/pull/10517))
-- feat: shed: incoming block-sub chainwatch tool ([filecoin-project/lotus#10513](https://github.com/filecoin-project/lotus/pull/10513))
-
-## Improvements
-- feat: chainstore: sharded mutex for filling chain height index
-- Check if epoch is negative in GetTipsetByHeight
-- fix: sched: Address GET_32G_MAX_CONCURRENT regression
-- fix: cli: Hide legacy markets cmds
-- fix: ci: Debugging m1 build
-- Disable lotus markets by default (#10809) ([filecoin-project/lotus#10809](https://github.com/filecoin-project/lotus/pull/10809))
-- perf: mempool: lower priority optimizations (#10693) ([filecoin-project/lotus#10693](https://github.com/filecoin-project/lotus/pull/10693))
-- perf: message pool: change locks to RWMutexes for performance ([filecoin-project/lotus#10561](https://github.com/filecoin-project/lotus/pull/10561))
-- perf: eth: gas estimate set applyTsMessages false (#10546) ([filecoin-project/lotus#10546](https://github.com/filecoin-project/lotus/pull/10546))
-- Change args check ([filecoin-project/lotus#10812](https://github.com/filecoin-project/lotus/pull/10812))
-- fix: sealing: Make lotus-worker report GPU usage to miner during ReplicaUpdate task (#10806) ([filecoin-project/lotus#10806](https://github.com/filecoin-project/lotus/pull/10806))
-- fix:splitstore:Don't block when potentially holding txnLk as writer ([filecoin-project/lotus#10811](https://github.com/filecoin-project/lotus/pull/10811))
-- fix: prover: Propagate skipped sectors in local PoSt
-- fix: unseal: check if sealed/update sector exists ([filecoin-project/lotus#10639](https://github.com/filecoin-project/lotus/pull/10639))
-- fix: sealing pipeline: Allow nil message in TerminateWait ([filecoin-project/lotus#10696](https://github.com/filecoin-project/lotus/pull/10696))
-- fix: cli: Check if the sectorID exists before removing ([filecoin-project/lotus#10611](https://github.com/filecoin-project/lotus/pull/10611))
-- feat:splitstore:limit moving gc threads ([filecoin-project/lotus#10621](https://github.com/filecoin-project/lotus/pull/10621))
-- fix: cli: Make `net connect` to miner address work ([filecoin-project/lotus#10599](https://github.com/filecoin-project/lotus/pull/10599))
-- fix: log: Stop logging `file does not exists` ([filecoin-project/lotus#10588](https://github.com/filecoin-project/lotus/pull/10588))
-- Update config default value (#10605) ([filecoin-project/lotus#10605](https://github.com/filecoin-project/lotus/pull/10605))
-- fix: cap the message gas limit at the block gas limit (#10637) ([filecoin-project/lotus#10637](https://github.com/filecoin-project/lotus/pull/10637))
-- fix: miner: correctly count sector extensions ([filecoin-project/lotus#10544](https://github.com/filecoin-project/lotus/pull/10544))
-- fix:mpool: prune excess messages before selection ([filecoin-project/lotus#10648](https://github.com/filecoin-project/lotus/pull/10648))
-- fix: proving: Initialize slice with with same length as partition ([filecoin-project/lotus#10569](https://github.com/filecoin-project/lotus/pull/10569))
-- perf: Address performance of EthGetTransactionCount  ([filecoin-project/lotus#10700](https://github.com/filecoin-project/lotus/pull/10700))
-- fix: sync: reduce log from error to info ([filecoin-project/lotus#10759](https://github.com/filecoin-project/lotus/pull/10759))
-- fix: state: lotus-miner info should show deals info without admin permission ([filecoin-project/lotus#10323](https://github.com/filecoin-project/lotus/pull/10323))
-- fix: tvx: make extract-multiple support the FVM ([filecoin-project/lotus#10714](https://github.com/filecoin-project/lotus/pull/10714))
-- feat: badger: add a has check before writing to reduce duplicates ([filecoin-project/lotus#10680](https://github.com/filecoin-project/lotus/pull/10680))
-- fix: chain: record heaviest tipset before notifying (#10694) ([filecoin-project/lotus#10694](https://github.com/filecoin-project/lotus/pull/10694))
-- fix: Eth JSON-RPC api: handle messages with gasFeeCap less than baseFee (#10614) ([filecoin-project/lotus#10614](https://github.com/filecoin-project/lotus/pull/10614))
-- feat: chainstore: optimize BlockMsgsForTipset ([filecoin-project/lotus#10552](https://github.com/filecoin-project/lotus/pull/10552))
-- refactor: stop using deprecated io/ioutil ([filecoin-project/lotus#10596](https://github.com/filecoin-project/lotus/pull/10596))
-- feat: shed: refactor market cron-state command ([filecoin-project/lotus#10746](https://github.com/filecoin-project/lotus/pull/10746))
-- fix: events: don't set GC confidence to 1 ([filecoin-project/lotus#10713](https://github.com/filecoin-project/lotus/pull/10713))
-- feat: sync: validate (early) that blocks fall within range (#10691) ([filecoin-project/lotus#10691](https://github.com/filecoin-project/lotus/pull/10691))
-- chainstore: Fix raw blocks getting scanned for links during snapshots (#10684) ([filecoin-project/lotus#10684](https://github.com/filecoin-project/lotus/pull/10684))
-- fix: remove pointless panic ([filecoin-project/lotus#10690](https://github.com/filecoin-project/lotus/pull/10690))
-- fix: check for nil bcastDict (#10646) ([filecoin-project/lotus#10646](https://github.com/filecoin-project/lotus/pull/10646))
-- fix: make state compute --html work with unknown methods ([filecoin-project/lotus#10619](https://github.com/filecoin-project/lotus/pull/10619))
-- shed: get balances of evm accounts  ([filecoin-project/lotus#10489](https://github.com/filecoin-project/lotus/pull/10489))
-- feat: Use MessageIndex in WaitForMessage ([filecoin-project/lotus#10587](https://github.com/filecoin-project/lotus/pull/10587))
-- fix: searchForIndexedMsg always returns an error ([filecoin-project/lotus#10586](https://github.com/filecoin-project/lotus/pull/10586))
-- Fix: export-range: Ignore ipld Blocks not found in Receipts. ([filecoin-project/lotus#10535](https://github.com/filecoin-project/lotus/pull/10535))
-- feat: stmgr: speed up calculation of genesis circ supply ([filecoin-project/lotus#10553](https://github.com/filecoin-project/lotus/pull/10553))
-- fix: gas estimation: don't special case paych collects ([filecoin-project/lotus#10549](https://github.com/filecoin-project/lotus/pull/10549))
-- fix: tracer: emit raw peer ids for compatibility with libp2p tracer ([filecoin-project/lotus#10271](https://github.com/filecoin-project/lotus/pull/10271))
-- Merge branch 'feat/new-gw-methods'
-
-## Dependencies
-- chore: deps: update to go-libp2p 0.27.5
-- devs: update libp2p #10937
-- chore: deps: update to FVM 3.3.1 ([filecoin-project/lotus#10786](https://github.com/filecoin-project/lotus/pull/10786))
-- chore: boxo: migrate from go-libipfs to boxo ([filecoin-project/lotus#10562](https://github.com/filecoin-project/lotus/pull/10562))
-- chore: deps: update to go-state-types v0.11.0-alpha-3 ([filecoin-project/lotus#10606](https://github.com/filecoin-project/lotus/pull/10606))
-- chore: bump go-libipfs ([filecoin-project/lotus#10531](https://github.com/filecoin-project/lotus/pull/10531))
-
-## Others
-- feat:networking:  (Synchronous) Consistent Broadcast for Filecoin EC ([filecoin-project/lotus#9858](https://github.com/filecoin-project/lotus/pull/9858))
-- Revert #9858 (consistent broadcast changes) ([filecoin-project/lotus#10777](https://github.com/filecoin-project/lotus/pull/10777))
-- Update build version for release/v1.23.1
-- chore: drop flaky TestBatchDealInput subcase ([filecoin-project/lotus#10810](https://github.com/filecoin-project/lotus/pull/10810))
-- chore: changelog clean up ([filecoin-project/lotus#10744](https://github.com/filecoin-project/lotus/pull/10744))
-- chore: refactor: drop unused IsTicketWinner (#10801) ([filecoin-project/lotus#10801](https://github.com/filecoin-project/lotus/pull/10801))
-- chore: build: bump matser version to v1.23.1-dev ([filecoin-project/lotus#10709](https://github.com/filecoin-project/lotus/pull/10709))
-- fix: deflake: use 2 miners for flaky tests ([filecoin-project/lotus#10764](https://github.com/filecoin-project/lotus/pull/10764))
-- test: eth: deflake multiblock lookup test (#10769) ([filecoin-project/lotus#10769](https://github.com/filecoin-project/lotus/pull/10769))
-- shed: migrations: add reminder about continuity testing tool ([filecoin-project/lotus#10762](https://github.com/filecoin-project/lotus/pull/10762))
-- chore: merge releases into master ([filecoin-project/lotus#10742](https://github.com/filecoin-project/lotus/pull/10742))
-- test: events: fix race when recording tipsets (#10665) ([filecoin-project/lotus#10665](https://github.com/filecoin-project/lotus/pull/10665))
-- fix: build: add CBDeliveryDelay to testground ([filecoin-project/lotus#10613](https://github.com/filecoin-project/lotus/pull/10613))
-- fix: build: Fixed incorrect words that could not be compiled ([filecoin-project/lotus#10610](https://github.com/filecoin-project/lotus/pull/10610))
-- build: docker: Update GO-version ([filecoin-project/lotus#10581](https://github.com/filecoin-project/lotus/pull/10581))
-- fix: itests: Don't call t.Error in MineBlocks goroutine ([filecoin-project/lotus#10572](https://github.com/filecoin-project/lotus/pull/10572))
-- docs: api: clarify MpoolClear params ([filecoin-project/lotus#10550](https://github.com/filecoin-project/lotus/pull/10550))
-
-Contributors
-
-| Contributor | Commits | Lines ± | Files Changed |
-|-------------|---------|---------|---------------|
-| vyzo | 70 | +1990/-429 | 135 |
-| Alfonso de la Rocha | 25 | +814/-299 | 56 |
-| Steven Allen | 14 | +125/-539 | 28 |
-| Shrenuj Bansal | 13 | +482/-138 | 52 |
-| Aayush | 17 | +317/-301 | 90 |
-| Łukasz Magiera | 13 | +564/-26 | 16 |
-| Jennifer Wang | 7 | +401/-140 | 10 |
-| Fridrik Asmundsson | 14 | +315/-84 | 20 |
-| Jorropo | 2 | +139/-137 | 74 |
-| Mikers | 6 | +114/-43 | 14 |
-| Hector Sanjuan | 5 | +92/-44 | 5 |
-| Ales Dumikau | 1 | +117/-0 | 10 |
-| Mike Seiler | 4 | +51/-51 | 6 |
-| zenground0 | 6 | +33/-25 | 8 |
-| Phi | 8 | +32/-10 | 10 |
-| Aayush Rajasekaran | 1 | +1/-32 | 2 |
-| Ian Davis | 2 | +7/-10 | 3 |
-| Marcel Telka | 1 | +5/-7 | 1 |
-| ychiao | 1 | +8/-3 | 2 |
-| jennijuju | 1 | +4/-4 | 8 |
-| adlrocha | 2 | +2/-2 | 2 |
-| Jiaying Wang | 1 | +0/-4 | 1 |
-| ZenGround0 | 1 | +2/-1 | 2 |
-| Zeng Li | 1 | +1/-1 | 1 |
+- feat: Added new environment variable `LOTUS_EXEC_TRACE_CACHE_SIZE` to configure execution trace cache size ([filecoin-project/lotus#10585](https://github.com/filecoin-project/lotus/pull/10585))
+  - If unset, we default to caching 16 most recent execution traces. Node operatores may want to set this to 0 while exchanges may want to crank it up. 
 
 # v1.23.0 / 2023-04-21
 

Dockerfile.lotus

@@ -1,273 +0,0 @@
-##### DEPRECATED
-
-FROM golang:1.19.7-buster AS builder-deps
-MAINTAINER Lotus Development Team
-
-RUN apt-get update && apt-get install -y ca-certificates build-essential clang ocl-icd-opencl-dev ocl-icd-libopencl1 jq libhwloc-dev
-
-ENV XDG_CACHE_HOME="/tmp"
-
-### taken from https://github.com/rust-lang/docker-rust/blob/master/1.63.0/buster/Dockerfile
-ENV RUSTUP_HOME=/usr/local/rustup \
-    CARGO_HOME=/usr/local/cargo \
-    PATH=/usr/local/cargo/bin:$PATH \
-    RUST_VERSION=1.63.0
-
-RUN set -eux; \
-    dpkgArch="$(dpkg --print-architecture)"; \
-    case "${dpkgArch##*-}" in \
-        amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='5cc9ffd1026e82e7fb2eec2121ad71f4b0f044e88bca39207b3f6b769aaa799c' ;; \
-        arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='e189948e396d47254103a49c987e7fb0e5dd8e34b200aa4481ecc4b8e41fb929' ;; \
-        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
-    esac; \
-    url="https://static.rust-lang.org/rustup/archive/1.25.1/${rustArch}/rustup-init"; \
-    wget "$url"; \
-    echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
-    chmod +x rustup-init; \
-    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
-    rm rustup-init; \
-    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
-    rustup --version; \
-    cargo --version; \
-    rustc --version;
-### end rust
-
-FROM builder-deps AS builder-local
-MAINTAINER Lotus Development Team
-
-COPY ./ /opt/filecoin
-WORKDIR /opt/filecoin
-
-### make configurable filecoin-ffi build
-ARG FFI_BUILD_FROM_SOURCE=0
-ENV FFI_BUILD_FROM_SOURCE=${FFI_BUILD_FROM_SOURCE}
-
-RUN make clean deps
-
-
-FROM builder-local AS builder-test
-MAINTAINER Lotus Development Team
-
-WORKDIR /opt/filecoin
-
-RUN make debug
-
-
-FROM builder-local AS builder
-MAINTAINER Lotus Development Team
-
-WORKDIR /opt/filecoin
-
-ARG RUSTFLAGS=""
-ARG GOFLAGS=""
-
-RUN make lotus lotus-miner lotus-worker lotus-shed lotus-wallet lotus-gateway lotus-stats
-
-
-FROM ubuntu:20.04 AS base
-MAINTAINER Lotus Development Team
-
-# Base resources
-COPY --from=builder /etc/ssl/certs                           /etc/ssl/certs
-COPY --from=builder /lib/*/libdl.so.2         /lib/
-COPY --from=builder /lib/*/librt.so.1         /lib/
-COPY --from=builder /lib/*/libgcc_s.so.1      /lib/
-COPY --from=builder /lib/*/libutil.so.1       /lib/
-COPY --from=builder /usr/lib/*/libltdl.so.7   /lib/
-COPY --from=builder /usr/lib/*/libnuma.so.1   /lib/
-COPY --from=builder /usr/lib/*/libhwloc.so.5  /lib/
-COPY --from=builder /usr/lib/*/libOpenCL.so.1 /lib/
-
-RUN useradd -r -u 532 -U fc \
- && mkdir -p /etc/OpenCL/vendors \
- && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
-
-###
-FROM base AS lotus
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-shed /usr/local/bin/
-COPY scripts/docker-lotus-entrypoint.sh /
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_PATH /var/lib/lotus
-ENV DOCKER_LOTUS_IMPORT_SNAPSHOT https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car
-ENV DOCKER_LOTUS_IMPORT_WALLET ""
-
-RUN mkdir /var/lib/lotus /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus /var/tmp/filecoin-proof-parameters
-
-VOLUME /var/lib/lotus
-VOLUME /var/tmp/filecoin-proof-parameters
-
-USER fc
-
-EXPOSE 1234
-
-ENTRYPOINT ["/docker-lotus-entrypoint.sh"]
-
-CMD ["-help"]
-
-###
-FROM base AS lotus-wallet
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-wallet /usr/local/bin/
-
-ENV WALLET_PATH /var/lib/lotus-wallet
-
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/lib/lotus-wallet
-
-VOLUME /var/lib/lotus-wallet
-
-USER fc
-
-EXPOSE 1777
-
-ENTRYPOINT ["/usr/local/bin/lotus-wallet"]
-
-CMD ["-help"]
-
-###
-FROM base AS lotus-gateway
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-gateway /usr/local/bin/
-
-USER fc
-
-EXPOSE 1234
-
-ENTRYPOINT ["/usr/local/bin/lotus-gateway"]
-
-CMD ["-help"]
-
-
-###
-FROM base AS lotus-miner
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-miner /usr/local/bin/
-COPY scripts/docker-lotus-miner-entrypoint.sh /
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-
-RUN mkdir /var/lib/lotus-miner /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus-miner /var/tmp/filecoin-proof-parameters
-
-VOLUME /var/lib/lotus-miner
-VOLUME /var/tmp/filecoin-proof-parameters
-
-USER fc
-
-EXPOSE 2345
-
-ENTRYPOINT ["/docker-lotus-miner-entrypoint.sh"]
-
-CMD ["-help"]
-
-
-###
-FROM base AS lotus-worker
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-worker /usr/local/bin/
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-
-RUN mkdir /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-worker
-
-VOLUME /var/lib/lotus-worker
-
-USER fc
-
-EXPOSE 3456
-
-ENTRYPOINT ["/usr/local/bin/lotus-worker"]
-
-CMD ["-help"]
-
-
-###
-from base as lotus-all-in-one
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-ENV LOTUS_PATH /var/lib/lotus
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-ENV WALLET_PATH /var/lib/lotus-wallet
-ENV DOCKER_LOTUS_IMPORT_SNAPSHOT https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car
-
-COPY --from=builder /opt/filecoin/lotus      /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-shed /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-wallet /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-gateway /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-miner /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-worker /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-stats /usr/local/bin/
-
-RUN mkdir /var/tmp/filecoin-proof-parameters
-RUN mkdir /var/lib/lotus
-RUN mkdir /var/lib/lotus-miner
-RUN mkdir /var/lib/lotus-worker
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus
-RUN chown fc: /var/lib/lotus-miner
-RUN chown fc: /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-wallet
-
-
-VOLUME /var/tmp/filecoin-proof-parameters
-VOLUME /var/lib/lotus
-VOLUME /var/lib/lotus-miner
-VOLUME /var/lib/lotus-worker
-VOLUME /var/lib/lotus-wallet
-
-EXPOSE 1234
-EXPOSE 2345
-EXPOSE 3456
-EXPOSE 1777
-
-###
-from base as lotus-test
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-ENV LOTUS_PATH /var/lib/lotus
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-ENV WALLET_PATH /var/lib/lotus-wallet
-
-COPY --from=builder-test /opt/filecoin/lotus      /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-miner /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-worker /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-seed /usr/local/bin/
-
-RUN mkdir /var/tmp/filecoin-proof-parameters
-RUN mkdir /var/lib/lotus
-RUN mkdir /var/lib/lotus-miner
-RUN mkdir /var/lib/lotus-worker
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus
-RUN chown fc: /var/lib/lotus-miner
-RUN chown fc: /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-wallet
-
-
-VOLUME /var/tmp/filecoin-proof-parameters
-VOLUME /var/lib/lotus
-VOLUME /var/lib/lotus-miner
-VOLUME /var/lib/lotus-worker
-VOLUME /var/lib/lotus-wallet
-
-EXPOSE 1234
-EXPOSE 2345
-EXPOSE 3456
-EXPOSE 1777
-

Makefile

@@ -355,7 +355,7 @@ fiximports:
 	./scripts/fiximports
 
 gen: actors-code-gen type-gen cfgdoc-gen docsgen api-gen circleci fiximports
-	@echo ">>> IF YOU'VE MODIFIED THE CLI OR CONFIG, REMEMBER TO ALSO MAKE docsgen-cli"
+	@echo ">>> IF YOU'VE MODIFIED THE CLI OR CONFIG, REMEMBER TO ALSO RUN 'make docsgen-cli'"
 .PHONY: gen
 
 jen: gen

api/api_full.go

@@ -812,6 +812,7 @@ type FullNode interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error) //perm:read
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)                              //perm:read
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)                                                                               //perm:read
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)                                                                        //perm:read
 	NetVersion(ctx context.Context) (string, error)                                                                                           //perm:read
 	NetListening(ctx context.Context) (bool, error)                                                                                           //perm:read
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)                                                                       //perm:read
@@ -829,23 +830,23 @@ type FullNode interface {
 
 	// Polling method for a filter, returns event logs which occurred since last poll.
 	// (requires write perm since timestamp of last filter execution will be written)
-	EthGetFilterChanges(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:write
+	EthGetFilterChanges(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:read
 
 	// Returns event logs matching filter with given id.
 	// (requires write perm since timestamp of last filter execution will be written)
-	EthGetFilterLogs(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:write
+	EthGetFilterLogs(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:read
 
 	// Installs a persistent filter based on given filter spec.
-	EthNewFilter(ctx context.Context, filter *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) //perm:write
+	EthNewFilter(ctx context.Context, filter *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) //perm:read
 
 	// Installs a persistent filter to notify when a new block arrives.
-	EthNewBlockFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:write
+	EthNewBlockFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:read
 
 	// Installs a persistent filter to notify when new messages arrive in the message pool.
-	EthNewPendingTransactionFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:write
+	EthNewPendingTransactionFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:read
 
 	// Uninstalls a filter with given id.
-	EthUninstallFilter(ctx context.Context, id ethtypes.EthFilterID) (bool, error) //perm:write
+	EthUninstallFilter(ctx context.Context, id ethtypes.EthFilterID) (bool, error) //perm:read
 
 	// Subscribe to different event types using websockets
 	// eventTypes is one or more of:
@@ -854,10 +855,10 @@ type FullNode interface {
 	//  - logs: notify new event logs that match a criteria
 	// params contains additional parameters used with the log event type
 	// The client will receive a stream of EthSubscriptionResponse values until EthUnsubscribe is called.
-	EthSubscribe(ctx context.Context, params jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) //perm:write
+	EthSubscribe(ctx context.Context, params jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) //perm:read
 
 	// Unsubscribe from a websocket subscription
-	EthUnsubscribe(ctx context.Context, id ethtypes.EthSubscriptionID) (bool, error) //perm:write
+	EthUnsubscribe(ctx context.Context, id ethtypes.EthSubscriptionID) (bool, error) //perm:read
 
 	// Returns the client version
 	Web3ClientVersion(ctx context.Context) (string, error) //perm:read

api/api_gateway.go

@@ -33,6 +33,9 @@ import (
 //  * Generate openrpc blobs
 
 type Gateway interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*MiningBaseInfo, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (MinerSectors, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*InvocResult, error)
@@ -98,6 +101,7 @@ type Gateway interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error)
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)
 	NetVersion(ctx context.Context) (string, error)
 	NetListening(ctx context.Context) (bool, error)
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)

api/docgen/docgen.go

@@ -86,6 +86,7 @@ func init() {
 	}
 
 	ExampleValues[reflect.TypeOf(addr)] = addr
+	ExampleValues[reflect.TypeOf(&addr)] = &addr
 
 	pid, err := peer.Decode("12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf")
 	if err != nil {

api/eth_aliases.go

@@ -21,6 +21,7 @@ func CreateEthRPCAliases(as apitypes.Aliaser) {
 	as.AliasMethod("eth_getStorageAt", "Filecoin.EthGetStorageAt")
 	as.AliasMethod("eth_getBalance", "Filecoin.EthGetBalance")
 	as.AliasMethod("eth_chainId", "Filecoin.EthChainId")
+	as.AliasMethod("eth_syncing", "Filecoin.EthSyncing")
 	as.AliasMethod("eth_feeHistory", "Filecoin.EthFeeHistory")
 	as.AliasMethod("eth_protocolVersion", "Filecoin.EthProtocolVersion")
 	as.AliasMethod("eth_maxPriorityFeePerGas", "Filecoin.EthMaxPriorityFeePerGas")

api/mocks/mock_full.go

@@ -1476,6 +1476,21 @@ func (mr *MockFullNodeMockRecorder) EthSubscribe(arg0, arg1 interface{}) *gomock
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EthSubscribe", reflect.TypeOf((*MockFullNode)(nil).EthSubscribe), arg0, arg1)
 }
 
+// EthSyncing mocks base method.
+func (m *MockFullNode) EthSyncing(arg0 context.Context) (ethtypes.EthSyncingResult, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "EthSyncing", arg0)
+	ret0, _ := ret[0].(ethtypes.EthSyncingResult)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// EthSyncing indicates an expected call of EthSyncing.
+func (mr *MockFullNodeMockRecorder) EthSyncing(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EthSyncing", reflect.TypeOf((*MockFullNode)(nil).EthSyncing), arg0)
+}
+
 // EthUninstallFilter mocks base method.
 func (m *MockFullNode) EthUninstallFilter(arg0 context.Context, arg1 ethtypes.EthFilterID) (bool, error) {
 	m.ctrl.T.Helper()

api/proxy_gen.go

@@ -274,9 +274,9 @@ type FullNodeMethods struct {
 
 	EthGetCode func(p0 context.Context, p1 ethtypes.EthAddress, p2 string) (ethtypes.EthBytes, error) `perm:"read"`
 
-	EthGetFilterChanges func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"write"`
+	EthGetFilterChanges func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
-	EthGetFilterLogs func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"write"`
+	EthGetFilterLogs func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
 	EthGetLogs func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
@@ -302,21 +302,23 @@ type FullNodeMethods struct {
 
 	EthMaxPriorityFeePerGas func(p0 context.Context) (ethtypes.EthBigInt, error) `perm:"read"`
 
-	EthNewBlockFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewBlockFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"read"`
 
-	EthNewFilter func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewFilter func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) `perm:"read"`
 
-	EthNewPendingTransactionFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewPendingTransactionFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"read"`
 
 	EthProtocolVersion func(p0 context.Context) (ethtypes.EthUint64, error) `perm:"read"`
 
 	EthSendRawTransaction func(p0 context.Context, p1 ethtypes.EthBytes) (ethtypes.EthHash, error) `perm:"read"`
 
-	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) `perm:"write"`
+	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) `perm:"read"`
 
-	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) `perm:"write"`
+	EthSyncing func(p0 context.Context) (ethtypes.EthSyncingResult, error) `perm:"read"`
 
-	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) `perm:"write"`
+	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) `perm:"read"`
+
+	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) `perm:"read"`
 
 	FilecoinAddressToEthAddress func(p0 context.Context, p1 address.Address) (ethtypes.EthAddress, error) `perm:"read"`
 
@@ -630,6 +632,8 @@ type GatewayStruct struct {
 }
 
 type GatewayMethods struct {
+	ChainGetBlock func(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) ``
+
 	ChainGetBlockMessages func(p0 context.Context, p1 cid.Cid) (*BlockMessages, error) ``
 
 	ChainGetGenesis func(p0 context.Context) (*types.TipSet, error) ``
@@ -722,6 +726,8 @@ type GatewayMethods struct {
 
 	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) ``
 
+	EthSyncing func(p0 context.Context) (ethtypes.EthSyncingResult, error) ``
+
 	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) ``
 
 	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) ``
@@ -730,8 +736,12 @@ type GatewayMethods struct {
 
 	GasEstimateMessageGas func(p0 context.Context, p1 *types.Message, p2 *MessageSendSpec, p3 types.TipSetKey) (*types.Message, error) ``
 
+	MinerGetBaseInfo func(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) ``
+
 	MpoolGetNonce func(p0 context.Context, p1 address.Address) (uint64, error) ``
 
+	MpoolPending func(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) ``
+
 	MpoolPush func(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) ``
 
 	MsigGetAvailableBalance func(p0 context.Context, p1 address.Address, p2 types.TipSetKey) (types.BigInt, error) ``
@@ -2422,6 +2432,17 @@ func (s *FullNodeStub) EthSubscribe(p0 context.Context, p1 jsonrpc.RawParams) (e
 	return *new(ethtypes.EthSubscriptionID), ErrNotSupported
 }
 
+func (s *FullNodeStruct) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	if s.Internal.EthSyncing == nil {
+		return *new(ethtypes.EthSyncingResult), ErrNotSupported
+	}
+	return s.Internal.EthSyncing(p0)
+}
+
+func (s *FullNodeStub) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	return *new(ethtypes.EthSyncingResult), ErrNotSupported
+}
+
 func (s *FullNodeStruct) EthUninstallFilter(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) {
 	if s.Internal.EthUninstallFilter == nil {
 		return false, ErrNotSupported
@@ -4094,6 +4115,17 @@ func (s *FullNodeStub) Web3ClientVersion(p0 context.Context) (string, error) {
 	return "", ErrNotSupported
 }
 
+func (s *GatewayStruct) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	if s.Internal.ChainGetBlock == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.ChainGetBlock(p0, p1)
+}
+
+func (s *GatewayStub) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) ChainGetBlockMessages(p0 context.Context, p1 cid.Cid) (*BlockMessages, error) {
 	if s.Internal.ChainGetBlockMessages == nil {
 		return nil, ErrNotSupported
@@ -4600,6 +4632,17 @@ func (s *GatewayStub) EthSubscribe(p0 context.Context, p1 jsonrpc.RawParams) (et
 	return *new(ethtypes.EthSubscriptionID), ErrNotSupported
 }
 
+func (s *GatewayStruct) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	if s.Internal.EthSyncing == nil {
+		return *new(ethtypes.EthSyncingResult), ErrNotSupported
+	}
+	return s.Internal.EthSyncing(p0)
+}
+
+func (s *GatewayStub) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	return *new(ethtypes.EthSyncingResult), ErrNotSupported
+}
+
 func (s *GatewayStruct) EthUninstallFilter(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) {
 	if s.Internal.EthUninstallFilter == nil {
 		return false, ErrNotSupported
@@ -4644,6 +4687,17 @@ func (s *GatewayStub) GasEstimateMessageGas(p0 context.Context, p1 *types.Messag
 	return nil, ErrNotSupported
 }
 
+func (s *GatewayStruct) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) {
+	if s.Internal.MinerGetBaseInfo == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.MinerGetBaseInfo(p0, p1, p2, p3)
+}
+
+func (s *GatewayStub) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolGetNonce(p0 context.Context, p1 address.Address) (uint64, error) {
 	if s.Internal.MpoolGetNonce == nil {
 		return 0, ErrNotSupported
@@ -4655,6 +4709,17 @@ func (s *GatewayStub) MpoolGetNonce(p0 context.Context, p1 address.Address) (uin
 	return 0, ErrNotSupported
 }
 
+func (s *GatewayStruct) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	if s.Internal.MpoolPending == nil {
+		return *new([]*types.SignedMessage), ErrNotSupported
+	}
+	return s.Internal.MpoolPending(p0, p1)
+}
+
+func (s *GatewayStub) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	return *new([]*types.SignedMessage), ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolPush(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) {
 	if s.Internal.MpoolPush == nil {
 		return *new(cid.Cid), ErrNotSupported

api/types.go

@@ -299,6 +299,7 @@ type MinerInfo struct {
 	SectorSize                 abi.SectorSize
 	WindowPoStPartitionSectors uint64
 	ConsensusFaultElapsed      abi.ChainEpoch
+	PendingOwnerAddress        *address.Address
 	Beneficiary                address.Address
 	BeneficiaryTerm            *miner.BeneficiaryTerm
 	PendingBeneficiaryTerm     *miner.PendingBeneficiaryChange
@@ -314,31 +315,30 @@ type NetworkParams struct {
 }
 
 type ForkUpgradeParams struct {
-	UpgradeSmokeHeight         abi.ChainEpoch
-	UpgradeBreezeHeight        abi.ChainEpoch
-	UpgradeIgnitionHeight      abi.ChainEpoch
-	UpgradeLiftoffHeight       abi.ChainEpoch
-	UpgradeAssemblyHeight      abi.ChainEpoch
-	UpgradeRefuelHeight        abi.ChainEpoch
-	UpgradeTapeHeight          abi.ChainEpoch
-	UpgradeKumquatHeight       abi.ChainEpoch
-	UpgradePriceListOopsHeight abi.ChainEpoch
-	BreezeGasTampingDuration   abi.ChainEpoch
-	UpgradeCalicoHeight        abi.ChainEpoch
-	UpgradePersianHeight       abi.ChainEpoch
-	UpgradeOrangeHeight        abi.ChainEpoch
-	UpgradeClausHeight         abi.ChainEpoch
-	UpgradeTrustHeight         abi.ChainEpoch
-	UpgradeNorwegianHeight     abi.ChainEpoch
-	UpgradeTurboHeight         abi.ChainEpoch
-	UpgradeHyperdriveHeight    abi.ChainEpoch
-	UpgradeChocolateHeight     abi.ChainEpoch
-	UpgradeOhSnapHeight        abi.ChainEpoch
-	UpgradeSkyrHeight          abi.ChainEpoch
-	UpgradeSharkHeight         abi.ChainEpoch
-	UpgradeHyggeHeight         abi.ChainEpoch
-	UpgradeLightningHeight     abi.ChainEpoch
-	UpgradeThunderHeight       abi.ChainEpoch
+	UpgradeSmokeHeight       abi.ChainEpoch
+	UpgradeBreezeHeight      abi.ChainEpoch
+	UpgradeIgnitionHeight    abi.ChainEpoch
+	UpgradeLiftoffHeight     abi.ChainEpoch
+	UpgradeAssemblyHeight    abi.ChainEpoch
+	UpgradeRefuelHeight      abi.ChainEpoch
+	UpgradeTapeHeight        abi.ChainEpoch
+	UpgradeKumquatHeight     abi.ChainEpoch
+	BreezeGasTampingDuration abi.ChainEpoch
+	UpgradeCalicoHeight      abi.ChainEpoch
+	UpgradePersianHeight     abi.ChainEpoch
+	UpgradeOrangeHeight      abi.ChainEpoch
+	UpgradeClausHeight       abi.ChainEpoch
+	UpgradeTrustHeight       abi.ChainEpoch
+	UpgradeNorwegianHeight   abi.ChainEpoch
+	UpgradeTurboHeight       abi.ChainEpoch
+	UpgradeHyperdriveHeight  abi.ChainEpoch
+	UpgradeChocolateHeight   abi.ChainEpoch
+	UpgradeOhSnapHeight      abi.ChainEpoch
+	UpgradeSkyrHeight        abi.ChainEpoch
+	UpgradeSharkHeight       abi.ChainEpoch
+	UpgradeHyggeHeight       abi.ChainEpoch
+	UpgradeLightningHeight   abi.ChainEpoch
+	UpgradeThunderHeight     abi.ChainEpoch
 }
 
 type NonceMapType map[address.Address]uint64

api/v0api/gateway.go

@@ -35,6 +35,9 @@ import (
 //  * Generate openrpc blobs
 
 type Gateway interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*api.MiningBaseInfo, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (api.MinerSectors, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*api.InvocResult, error)

api/v0api/proxy_gen.go

@@ -431,6 +431,8 @@ type GatewayStruct struct {
 }
 
 type GatewayMethods struct {
+	ChainGetBlock func(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) ``
+
 	ChainGetBlockMessages func(p0 context.Context, p1 cid.Cid) (*api.BlockMessages, error) ``
 
 	ChainGetMessage func(p0 context.Context, p1 cid.Cid) (*types.Message, error) ``
@@ -453,8 +455,12 @@ type GatewayMethods struct {
 
 	GasEstimateMessageGas func(p0 context.Context, p1 *types.Message, p2 *api.MessageSendSpec, p3 types.TipSetKey) (*types.Message, error) ``
 
+	MinerGetBaseInfo func(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) ``
+
 	MpoolGetNonce func(p0 context.Context, p1 address.Address) (uint64, error) ``
 
+	MpoolPending func(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) ``
+
 	MpoolPush func(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) ``
 
 	MsigGetAvailableBalance func(p0 context.Context, p1 address.Address, p2 types.TipSetKey) (types.BigInt, error) ``
@@ -2581,6 +2587,17 @@ func (s *FullNodeStub) WalletVerify(p0 context.Context, p1 address.Address, p2 [
 	return false, ErrNotSupported
 }
 
+func (s *GatewayStruct) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	if s.Internal.ChainGetBlock == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.ChainGetBlock(p0, p1)
+}
+
+func (s *GatewayStub) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) ChainGetBlockMessages(p0 context.Context, p1 cid.Cid) (*api.BlockMessages, error) {
 	if s.Internal.ChainGetBlockMessages == nil {
 		return nil, ErrNotSupported
@@ -2702,6 +2719,17 @@ func (s *GatewayStub) GasEstimateMessageGas(p0 context.Context, p1 *types.Messag
 	return nil, ErrNotSupported
 }
 
+func (s *GatewayStruct) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) {
+	if s.Internal.MinerGetBaseInfo == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.MinerGetBaseInfo(p0, p1, p2, p3)
+}
+
+func (s *GatewayStub) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolGetNonce(p0 context.Context, p1 address.Address) (uint64, error) {
 	if s.Internal.MpoolGetNonce == nil {
 		return 0, ErrNotSupported
@@ -2713,6 +2741,17 @@ func (s *GatewayStub) MpoolGetNonce(p0 context.Context, p1 address.Address) (uin
 	return 0, ErrNotSupported
 }
 
+func (s *GatewayStruct) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	if s.Internal.MpoolPending == nil {
+		return *new([]*types.SignedMessage), ErrNotSupported
+	}
+	return s.Internal.MpoolPending(p0, p1)
+}
+
+func (s *GatewayStub) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	return *new([]*types.SignedMessage), ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolPush(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) {
 	if s.Internal.MpoolPush == nil {
 		return *new(cid.Cid), ErrNotSupported

blockstore/badger/blockstore.go

@@ -660,6 +660,60 @@ func (b *Blockstore) Has(ctx context.Context, cid cid.Cid) (bool, error) {
 	}
 }
 
+// GetMany implements a GetMany method for the badgerDB blockstore
+// This method returns empty slices and an error if an error other than ErrKeyNotFound is encountered
+// For every ErrKeyNotFound, the missing CID is returned but the rest of the results are returned without error
+func (b *Blockstore) GetMany(ctx context.Context, cids []cid.Cid) ([]blocks.Block, []cid.Cid, error) {
+	if err := b.access(); err != nil {
+		return nil, nil, err
+	}
+	defer b.viewers.Done()
+
+	b.lockDB()
+	defer b.unlockDB()
+
+	blks := make([]blocks.Block, 0, len(cids))
+	missingCids := make([]cid.Cid, 0, len(cids))
+	err := b.db.View(func(txn *badger.Txn) error {
+		f := func(c cid.Cid) error { // encapsulate in an anonymous function to handle key pool deferral for each key
+			if !c.Defined() {
+				return ipld.ErrNotFound{Cid: c}
+			}
+			k, pooled := b.PooledStorageKey(c)
+			if pooled {
+				defer KeyPool.Put(k)
+			}
+			switch item, err := txn.Get(k); err {
+			case nil:
+				val, err := item.ValueCopy(nil)
+				if err != nil {
+					return err
+				}
+				blk, err := blocks.NewBlockWithCid(val, c)
+				if err != nil {
+					return err
+				}
+				blks = append(blks, blk)
+			case badger.ErrKeyNotFound:
+				missingCids = append(missingCids, c)
+			default:
+				return fmt.Errorf("failed to get block from badger blockstore: %w", err)
+			}
+			return nil
+		}
+		for _, c := range cids {
+			if err := f(c); err != nil {
+				return err
+			}
+		}
+		return nil
+	})
+	if err != nil {
+		return nil, nil, err
+	}
+	return blks, missingCids, nil
+}
+
 // Get implements Blockstore.Get.
 func (b *Blockstore) Get(ctx context.Context, cid cid.Cid) (blocks.Block, error) {
 	if !cid.Defined() {

blockstore/splitstore/splitstore.go

@@ -164,7 +164,7 @@ type SplitStore struct {
 	path string
 
 	mx          sync.Mutex
-	warmupEpoch abi.ChainEpoch // protected by mx
+	warmupEpoch atomic.Int64
 	baseEpoch   abi.ChainEpoch // protected by compaction lock
 	pruneEpoch  abi.ChainEpoch // protected by compaction lock
 
@@ -684,9 +684,7 @@ func (s *SplitStore) View(ctx context.Context, cid cid.Cid, cb func([]byte) erro
 }
 
 func (s *SplitStore) isWarm() bool {
-	s.mx.Lock()
-	defer s.mx.Unlock()
-	return s.warmupEpoch > 0
+	return s.warmupEpoch.Load() > 0
 }
 
 // State tracking
@@ -757,7 +755,7 @@ func (s *SplitStore) Start(chain ChainAccessor, us stmgr.UpgradeSchedule) error
 	bs, err = s.ds.Get(s.ctx, warmupEpochKey)
 	switch err {
 	case nil:
-		s.warmupEpoch = bytesToEpoch(bs)
+		s.warmupEpoch.Store(bytesToInt64(bs))
 
 	case dstore.ErrNotFound:
 		warmup = true
@@ -791,7 +789,7 @@ func (s *SplitStore) Start(chain ChainAccessor, us stmgr.UpgradeSchedule) error
 		return xerrors.Errorf("error loading compaction index: %w", err)
 	}
 
-	log.Infow("starting splitstore", "baseEpoch", s.baseEpoch, "warmupEpoch", s.warmupEpoch)
+	log.Infow("starting splitstore", "baseEpoch", s.baseEpoch, "warmupEpoch", s.warmupEpoch.Load())
 
 	if warmup {
 		err = s.warmup(curTs)

blockstore/splitstore/splitstore_check.go

@@ -145,7 +145,7 @@ func (s *SplitStore) doCheck(curTs *types.TipSet) error {
 func (s *SplitStore) Info() map[string]interface{} {
 	info := make(map[string]interface{})
 	info["base epoch"] = s.baseEpoch
-	info["warmup epoch"] = s.warmupEpoch
+	info["warmup epoch"] = s.warmupEpoch.Load()
 	info["compactions"] = s.compactionIndex
 	info["prunes"] = s.pruneIndex
 	info["compacting"] = s.compacting == 1

blockstore/splitstore/splitstore_compact.go

@@ -1114,13 +1114,17 @@ func (s *SplitStore) walkChain(ts *types.TipSet, inclState, inclMsgs abi.ChainEp
 					if err := walkBlock(c); err != nil {
 						return xerrors.Errorf("error walking block (cid: %s): %w", c, err)
 					}
+
+					if err := s.checkYield(); err != nil {
+						return xerrors.Errorf("check yield: %w", err)
+					}
 				}
 				return nil
 			})
 		}
 
 		if err := g.Wait(); err != nil {
-			return err
+			return xerrors.Errorf("walkBlock workers errored: %w", err)
 		}
 	}
 
@@ -1153,8 +1157,8 @@ func (s *SplitStore) walkObject(c cid.Cid, visitor ObjectVisitor, f func(cid.Cid
 	}
 
 	// check this before recursing
-	if err := s.checkYield(); err != nil {
-		return 0, err
+	if err := s.checkClosing(); err != nil {
+		return 0, xerrors.Errorf("check closing: %w", err)
 	}
 
 	var links []cid.Cid
@@ -1222,8 +1226,8 @@ func (s *SplitStore) walkObjectIncomplete(c cid.Cid, visitor ObjectVisitor, f, m
 	}
 
 	// check this before recursing
-	if err := s.checkYield(); err != nil {
-		return sz, err
+	if err := s.checkClosing(); err != nil {
+		return sz, xerrors.Errorf("check closing: %w", err)
 	}
 
 	var links []cid.Cid

blockstore/splitstore/splitstore_test.go

@@ -429,7 +429,7 @@ func testSplitStoreReification(t *testing.T, f func(context.Context, blockstore.
 	}
 	defer ss.Close() //nolint
 
-	ss.warmupEpoch = 1
+	ss.warmupEpoch.Store(1)
 	go ss.reifyOrchestrator()
 
 	waitForReification := func() {
@@ -529,7 +529,7 @@ func testSplitStoreReificationLimit(t *testing.T, f func(context.Context, blocks
 	}
 	defer ss.Close() //nolint
 
-	ss.warmupEpoch = 1
+	ss.warmupEpoch.Store(1)
 	go ss.reifyOrchestrator()
 
 	waitForReification := func() {

blockstore/splitstore/splitstore_warmup.go

@@ -136,9 +136,8 @@ func (s *SplitStore) doWarmup(curTs *types.TipSet) error {
 	if err != nil {
 		return xerrors.Errorf("error saving warm up epoch: %w", err)
 	}
-	s.mx.Lock()
-	s.warmupEpoch = epoch
-	s.mx.Unlock()
+
+	s.warmupEpoch.Store(int64(epoch))
 
 	// also save the compactionIndex, as this is used as an indicator of warmup for upgraded nodes
 	err = s.ds.Put(s.ctx, compactionIndexKey, int64ToBytes(s.compactionIndex))

build/params_interop.go

@@ -49,16 +49,11 @@ var UpgradeHyperdriveHeight = abi.ChainEpoch(-16)
 var UpgradeChocolateHeight = abi.ChainEpoch(-17)
 var UpgradeOhSnapHeight = abi.ChainEpoch(-18)
 var UpgradeSkyrHeight = abi.ChainEpoch(-19)
+var UpgradeSharkHeight = abi.ChainEpoch(-20)
+var UpgradeHyggeHeight = abi.ChainEpoch(-21)
+var UpgradeLightningHeight = abi.ChainEpoch(-22)
 
-const UpgradeSharkHeight = abi.ChainEpoch(-20)
-
-const UpgradeHyggeHeight = abi.ChainEpoch(100)
-
-// ??????????
-const UpgradeLightningHeight = 200
-
-// ??????????????????
-const UpgradeThunderHeight = 300
+const UpgradeThunderHeight = 50
 
 var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
 	0: DrandMainnet,

build/version.go

@@ -37,7 +37,7 @@ func BuildTypeString() string {
 }
 
 // BuildVersion is the local build version
-const BuildVersion = "1.23.1-rc4"
+const BuildVersion = "1.23.2-dev"
 
 func UserVersion() string {
 	if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {

chain/actors/policy/policy.go

@@ -61,6 +61,10 @@ const (
 	MaxPreCommitRandomnessLookback = builtin11.EpochsInDay + SealRandomnessLookback
 )
 
+var (
+	MarketDefaultAllocationTermBuffer = market11.MarketDefaultAllocationTermBuffer
+)
+
 // SetSupportedProofTypes sets supported proof types, across all actor versions.
 // This should only be used for testing.
 func SetSupportedProofTypes(types ...abi.RegisteredSealProof) {

chain/actors/policy/policy.go.template

@@ -39,6 +39,10 @@ const (
 	MaxPreCommitRandomnessLookback = builtin{{.latestVersion}}.EpochsInDay + SealRandomnessLookback
 )
 
+var (
+	MarketDefaultAllocationTermBuffer = market{{.latestVersion}}.MarketDefaultAllocationTermBuffer
+)
+
 // SetSupportedProofTypes sets supported proof types, across all actor versions.
 // This should only be used for testing.
 func SetSupportedProofTypes(types ...abi.RegisteredSealProof) {

chain/events/observer.go

@@ -125,7 +125,7 @@ func (o *observer) listenHeadChangesOnce(ctx context.Context) error {
 
 	for changes := range notifs {
 		if err := o.applyChanges(ctx, changes); err != nil {
-			return err
+			return xerrors.Errorf("failed to apply a change notification: %w", err)
 		}
 	}
 

chain/gen/slashfilter/slashfilter.go

@@ -11,7 +11,6 @@ import (
 
 	"github.com/filecoin-project/go-state-types/abi"
 
-	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
 )
 
@@ -27,24 +26,20 @@ func New(dstore ds.Batching) *SlashFilter {
 	}
 }
 
-func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, parentEpoch abi.ChainEpoch) error {
-	if build.IsNearUpgrade(bh.Height, build.UpgradeOrangeHeight) {
-		return nil
-	}
-
+func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, parentEpoch abi.ChainEpoch) (cid.Cid, error) {
 	epochKey := ds.NewKey(fmt.Sprintf("/%s/%d", bh.Miner, bh.Height))
 	{
 		// double-fork mining (2 blocks at one epoch)
-		if err := checkFault(ctx, f.byEpoch, epochKey, bh, "double-fork mining faults"); err != nil {
-			return err
+		if witness, err := checkFault(ctx, f.byEpoch, epochKey, bh, "double-fork mining faults"); err != nil {
+			return witness, xerrors.Errorf("check double-fork mining faults: %w", err)
 		}
 	}
 
 	parentsKey := ds.NewKey(fmt.Sprintf("/%s/%x", bh.Miner, types.NewTipSetKey(bh.Parents...).Bytes()))
 	{
 		// time-offset mining faults (2 blocks with the same parents)
-		if err := checkFault(ctx, f.byParents, parentsKey, bh, "time-offset mining faults"); err != nil {
-			return err
+		if witness, err := checkFault(ctx, f.byParents, parentsKey, bh, "time-offset mining faults"); err != nil {
+			return witness, xerrors.Errorf("check time-offset mining faults: %w", err)
 		}
 	}
 
@@ -55,19 +50,19 @@ func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, par
 		parentEpochKey := ds.NewKey(fmt.Sprintf("/%s/%d", bh.Miner, parentEpoch))
 		have, err := f.byEpoch.Has(ctx, parentEpochKey)
 		if err != nil {
-			return err
+			return cid.Undef, err
 		}
 
 		if have {
 			// If we had, make sure it's in our parent tipset
 			cidb, err := f.byEpoch.Get(ctx, parentEpochKey)
 			if err != nil {
-				return xerrors.Errorf("getting other block cid: %w", err)
+				return cid.Undef, xerrors.Errorf("getting other block cid: %w", err)
 			}
 
 			_, parent, err := cid.CidFromBytes(cidb)
 			if err != nil {
-				return err
+				return cid.Undef, err
 			}
 
 			var found bool
@@ -78,45 +73,45 @@ func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, par
 			}
 
 			if !found {
-				return xerrors.Errorf("produced block would trigger 'parent-grinding fault' consensus fault; miner: %s; bh: %s, expected parent: %s", bh.Miner, bh.Cid(), parent)
+				return parent, xerrors.Errorf("produced block would trigger 'parent-grinding fault' consensus fault; miner: %s; bh: %s, expected parent: %s", bh.Miner, bh.Cid(), parent)
 			}
 		}
 	}
 
 	if err := f.byParents.Put(ctx, parentsKey, bh.Cid().Bytes()); err != nil {
-		return xerrors.Errorf("putting byEpoch entry: %w", err)
+		return cid.Undef, xerrors.Errorf("putting byEpoch entry: %w", err)
 	}
 
 	if err := f.byEpoch.Put(ctx, epochKey, bh.Cid().Bytes()); err != nil {
-		return xerrors.Errorf("putting byEpoch entry: %w", err)
+		return cid.Undef, xerrors.Errorf("putting byEpoch entry: %w", err)
 	}
 
-	return nil
+	return cid.Undef, nil
 }
 
-func checkFault(ctx context.Context, t ds.Datastore, key ds.Key, bh *types.BlockHeader, faultType string) error {
+func checkFault(ctx context.Context, t ds.Datastore, key ds.Key, bh *types.BlockHeader, faultType string) (cid.Cid, error) {
 	fault, err := t.Has(ctx, key)
 	if err != nil {
-		return err
+		return cid.Undef, xerrors.Errorf("failed to read from datastore: %w", err)
 	}
 
 	if fault {
 		cidb, err := t.Get(ctx, key)
 		if err != nil {
-			return xerrors.Errorf("getting other block cid: %w", err)
+			return cid.Undef, xerrors.Errorf("getting other block cid: %w", err)
 		}
 
 		_, other, err := cid.CidFromBytes(cidb)
 		if err != nil {
-			return err
+			return cid.Undef, err
 		}
 
 		if other == bh.Cid() {
-			return nil
+			return cid.Undef, nil
 		}
 
-		return xerrors.Errorf("produced block would trigger '%s' consensus fault; miner: %s; bh: %s, other: %s", faultType, bh.Miner, bh.Cid(), other)
+		return other, xerrors.Errorf("produced block would trigger '%s' consensus fault; miner: %s; bh: %s, other: %s", faultType, bh.Miner, bh.Cid(), other)
 	}
 
-	return nil
+	return cid.Undef, nil
 }

chain/index/msgindex.go

@@ -37,7 +37,17 @@ var dbDefs = []string{
 	)`,
 	`INSERT OR IGNORE INTO _meta (version) VALUES (1)`,
 }
-var dbPragmas = []string{}
+
+var dbPragmas = []string{
+	"PRAGMA synchronous = normal",
+	"PRAGMA temp_store = memory",
+	"PRAGMA mmap_size = 30000000000",
+	"PRAGMA page_size = 32768",
+	"PRAGMA auto_vacuum = NONE",
+	"PRAGMA automatic_index = OFF",
+	"PRAGMA journal_mode = WAL",
+	"PRAGMA read_uncommitted = ON",
+}
 
 const (
 	// prepared stmts

chain/index/msgindex_test.go

@@ -39,10 +39,10 @@ func TestBasicMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	t.Log("verifying index")
 	verifyIndex(t, cs, msgIndex)
 }
@@ -51,7 +51,7 @@ func TestReorgMsgIndex(t *testing.T) {
 	// slightly more nuanced test that includes reorgs
 	// 1. Create an index with mock chain store
 	// 2. Advance/Reorg the chain for a few tipsets
-	// 3. Verify that the index contains all messages with the correct tipst/epoch
+	// 3. Verify that the index contains all messages with the correct tipset/epoch
 	cs := newMockChainStore()
 	cs.genesis()
 
@@ -67,10 +67,10 @@ func TestReorgMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	// a simple reorg
 	t.Log("doing reorg")
 	reorgme := cs.curTs
@@ -80,7 +80,8 @@ func TestReorgMsgIndex(t *testing.T) {
 	reorgmeChild := cs.makeBlk()
 	err = cs.reorg([]*types.TipSet{reorgme}, []*types.TipSet{reorgmeChild})
 	require.NoError(t, err)
-	time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
+
+	waitForCoalescerAfterLastEvent()
 
 	t.Log("verifying index")
 	verifyIndex(t, cs, msgIndex)
@@ -109,10 +110,10 @@ func TestReconcileMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	// Close it and reorg
 	err = msgIndex.Close()
 	require.NoError(t, err)
@@ -296,3 +297,11 @@ func (cs *mockChainStore) GetTipSetFromKey(ctx context.Context, tsk types.TipSet
 	}
 	return ts, nil
 }
+
+func waitForCoalescerAfterLastEvent() {
+	// It can take up to CoalesceMinDelay for the coalescer timer to fire after the last event.
+	// When the timer fires, it can wait up to CoalesceMinDelay again for more events.
+	// Therefore the total wait is 2 * CoalesceMinDelay.
+	// Then we wait another second for the listener (the index) to actually process events.
+	time.Sleep(2*CoalesceMinDelay + time.Second)
+}

chain/messagepool/check.go

@@ -35,8 +35,8 @@ func (mp *MessagePool) CheckPendingMessages(ctx context.Context, from address.Ad
 	mp.lk.RLock()
 	mset, ok, err := mp.getPendingMset(ctx, from)
 	if err != nil {
-		log.Warnf("errored while getting pending mset: %w", err)
-		return nil, err
+		mp.lk.RUnlock()
+		return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 	}
 	if ok {
 		msgs = make([]*types.Message, 0, len(mset.msgs))
@@ -71,8 +71,8 @@ func (mp *MessagePool) CheckReplaceMessages(ctx context.Context, replace []*type
 			msgMap[m.From] = mmap
 			mset, ok, err := mp.getPendingMset(ctx, m.From)
 			if err != nil {
-				log.Warnf("errored while getting pending mset: %w", err)
-				return nil, err
+				mp.lk.RUnlock()
+				return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 			}
 			if ok {
 				count += len(mset.msgs)
@@ -155,8 +155,8 @@ func (mp *MessagePool) checkMessages(ctx context.Context, msgs []*types.Message,
 			mp.lk.RLock()
 			mset, ok, err := mp.getPendingMset(ctx, m.From)
 			if err != nil {
-				log.Warnf("errored while getting pending mset: %w", err)
-				return nil, err
+				mp.lk.RUnlock()
+				return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 			}
 			if ok && !interned {
 				st = &actorState{nextNonce: mset.nextNonce, requiredFunds: mset.requiredFunds}

chain/messagepool/messagepool.go

@@ -448,12 +448,8 @@ func New(ctx context.Context, api Provider, ds dtypes.MetadataDS, us stmgr.Upgra
 	return mp, nil
 }
 
-func (mp *MessagePool) TryForEachPendingMessage(f func(cid.Cid) error) error {
-	// avoid deadlocks in splitstore compaction when something else needs to access the blockstore
-	// while holding the mpool lock
-	if !mp.lk.TryLock() {
-		return xerrors.Errorf("mpool TryForEachPendingMessage: could not acquire lock")
-	}
+func (mp *MessagePool) ForEachPendingMessage(f func(cid.Cid) error) error {
+	mp.lk.Lock()
 	defer mp.lk.Unlock()
 
 	for _, mset := range mp.pending {
@@ -749,8 +745,7 @@ func (mp *MessagePool) checkMessage(ctx context.Context, m *types.SignedMessage)
 	}
 
 	if err := mp.VerifyMsgSig(m); err != nil {
-		log.Warnf("signature verification failed: %s", err)
-		return err
+		return xerrors.Errorf("signature verification failed: %s", err)
 	}
 
 	return nil
@@ -969,13 +964,11 @@ func (mp *MessagePool) addLocked(ctx context.Context, m *types.SignedMessage, st
 	}
 
 	if _, err := mp.api.PutMessage(ctx, m); err != nil {
-		log.Warnf("mpooladd cs.PutMessage failed: %s", err)
-		return err
+		return xerrors.Errorf("mpooladd cs.PutMessage failed: %s", err)
 	}
 
 	if _, err := mp.api.PutMessage(ctx, &m.Message); err != nil {
-		log.Warnf("mpooladd cs.PutMessage failed: %s", err)
-		return err
+		return xerrors.Errorf("mpooladd cs.PutMessage failed: %s", err)
 	}
 
 	// Note: If performance becomes an issue, making this getOrCreatePendingMset will save some work

chain/messagepool/messagepool_test.go

@@ -11,9 +11,11 @@ import (
 	"github.com/ipfs/go-datastore"
 	logging "github.com/ipfs/go-log/v2"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
+	big2 "github.com/filecoin-project/go-state-types/big"
 	"github.com/filecoin-project/go-state-types/crypto"
 	"github.com/filecoin-project/go-state-types/network"
 	builtin2 "github.com/filecoin-project/specs-actors/v2/actors/builtin"
@@ -524,6 +526,36 @@ func TestPruningSimple(t *testing.T) {
 	}
 }
 
+func TestGasRewardNegative(t *testing.T) {
+	var mp MessagePool
+
+	msg := types.SignedMessage{
+		Message: types.Message{
+			GasLimit:   1000,
+			GasFeeCap:  big2.NewInt(20000),
+			GasPremium: big2.NewInt(15000),
+		},
+	}
+	baseFee := big2.NewInt(30000)
+	// Over the GasPremium, but under the BaseFee
+	gr1 := mp.getGasReward(&msg, baseFee)
+
+	msg.Message.GasFeeCap = big2.NewInt(15000)
+	// Equal to GasPremium, under the BaseFee
+	gr2 := mp.getGasReward(&msg, baseFee)
+
+	msg.Message.GasFeeCap = big2.NewInt(10000)
+	// Under both GasPremium and BaseFee
+	gr3 := mp.getGasReward(&msg, baseFee)
+
+	require.True(t, gr1.Sign() < 0)
+	require.True(t, gr2.Sign() < 0)
+	require.True(t, gr3.Sign() < 0)
+
+	require.True(t, gr1.Cmp(gr2) > 0)
+	require.True(t, gr2.Cmp(gr3) > 0)
+}
+
 func TestLoadLocal(t *testing.T) {
 	tma := newTestMpoolAPI()
 	ds := datastore.NewMapDatastore()

chain/messagepool/selection_test.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"sort"
 	"testing"
+	"time"
 
 	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-datastore"
@@ -1690,3 +1691,188 @@ readLoop:
 	}
 
 }
+
+func TestRealWorldSelectionTiming(t *testing.T) {
+	//stm: @TOKEN_WALLET_NEW_001, @TOKEN_WALLET_SIGN_001, @CHAIN_MEMPOOL_SELECT_001
+
+	// load test-messages.json.gz and rewrite the messages so that
+	// 1) we map each real actor to a test actor so that we can sign the messages
+	// 2) adjust the nonces so that they start from 0
+	file, err := os.Open("test-messages2.json.gz")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	gzr, err := gzip.NewReader(file)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dec := json.NewDecoder(gzr)
+
+	var msgs []*types.SignedMessage
+	baseNonces := make(map[address.Address]uint64)
+
+readLoop:
+	for {
+		m := new(types.SignedMessage)
+		err := dec.Decode(m)
+		switch err {
+		case nil:
+			msgs = append(msgs, m)
+			nonce, ok := baseNonces[m.Message.From]
+			if !ok || m.Message.Nonce < nonce {
+				baseNonces[m.Message.From] = m.Message.Nonce
+			}
+
+		case io.EOF:
+			break readLoop
+
+		default:
+			t.Fatal(err)
+		}
+	}
+
+	actorMap := make(map[address.Address]address.Address)
+	actorWallets := make(map[address.Address]api.Wallet)
+
+	for _, m := range msgs {
+		baseNonce := baseNonces[m.Message.From]
+
+		localActor, ok := actorMap[m.Message.From]
+		if !ok {
+			w, err := wallet.NewWallet(wallet.NewMemKeyStore())
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			a, err := w.WalletNew(context.Background(), types.KTSecp256k1)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			actorMap[m.Message.From] = a
+			actorWallets[a] = w
+			localActor = a
+		}
+
+		w, ok := actorWallets[localActor]
+		if !ok {
+			t.Fatalf("failed to lookup wallet for actor %s", localActor)
+		}
+
+		m.Message.From = localActor
+		m.Message.Nonce -= baseNonce
+
+		sig, err := w.WalletSign(context.TODO(), localActor, m.Message.Cid().Bytes(), api.MsgMeta{})
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		m.Signature = *sig
+	}
+
+	mp, tma := makeTestMpool()
+
+	block := tma.nextBlockWithHeight(uint64(build.UpgradeHyggeHeight) + 10)
+	ts := mock.TipSet(block)
+	tma.applyBlock(t, block)
+
+	for _, a := range actorMap {
+		tma.setBalance(a, 1000000)
+	}
+
+	tma.baseFee = types.NewInt(800_000_000)
+
+	sort.Slice(msgs, func(i, j int) bool {
+		return msgs[i].Message.Nonce < msgs[j].Message.Nonce
+	})
+
+	// add the messages
+	for _, m := range msgs {
+		mustAdd(t, mp, m)
+	}
+
+	// do message selection and check block packing
+	minGasLimit := int64(0.9 * float64(build.BlockGasLimit))
+
+	// greedy first
+	start := time.Now()
+	selected, err := mp.SelectMessages(context.Background(), ts, 1.0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit := int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=1.0; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// high quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .8)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.8; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// mid quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .4)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.4; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// low quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.1; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// very low quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .01)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.01; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+}

chain/stmgr/execute.go

@@ -131,15 +131,17 @@ func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (c
 	tsKey := ts.Key()
 
 	// check if we have the trace for this tipset in the cache
-	sm.execTraceCacheLock.Lock()
-	if entry, ok := sm.execTraceCache.Get(tsKey); ok {
-		// we have to make a deep copy since caller can modify the invocTrace
-		// and we don't want that to change what we store in cache
-		invocTraceCopy := makeDeepCopy(entry.invocTrace)
+	if execTraceCacheSize > 0 {
+		sm.execTraceCacheLock.Lock()
+		if entry, ok := sm.execTraceCache.Get(tsKey); ok {
+			// we have to make a deep copy since caller can modify the invocTrace
+			// and we don't want that to change what we store in cache
+			invocTraceCopy := makeDeepCopy(entry.invocTrace)
+			sm.execTraceCacheLock.Unlock()
+			return entry.postStateRoot, invocTraceCopy, nil
+		}
 		sm.execTraceCacheLock.Unlock()
-		return entry.postStateRoot, invocTraceCopy, nil
 	}
-	sm.execTraceCacheLock.Unlock()
 
 	var invocTrace []*api.InvocResult
 	st, err := sm.ExecutionTraceWithMonitor(ctx, ts, &InvocationTracer{trace: &invocTrace})
@@ -147,11 +149,13 @@ func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (c
 		return cid.Undef, nil, err
 	}
 
-	invocTraceCopy := makeDeepCopy(invocTrace)
+	if execTraceCacheSize > 0 {
+		invocTraceCopy := makeDeepCopy(invocTrace)
 
-	sm.execTraceCacheLock.Lock()
-	sm.execTraceCache.Add(tsKey, tipSetCacheEntry{st, invocTraceCopy})
-	sm.execTraceCacheLock.Unlock()
+		sm.execTraceCacheLock.Lock()
+		sm.execTraceCache.Add(tsKey, tipSetCacheEntry{st, invocTraceCopy})
+		sm.execTraceCacheLock.Unlock()
+	}
 
 	return st, invocTrace, nil
 }

chain/stmgr/stmgr.go

@@ -3,6 +3,8 @@ package stmgr
 import (
 	"context"
 	"fmt"
+	"os"
+	"strconv"
 	"sync"
 
 	lru "github.com/hashicorp/golang-lru/v2"
@@ -40,8 +42,7 @@ import (
 const LookbackNoLimit = api.LookbackNoLimit
 const ReceiptAmtBitwidth = 3
 
-const execTraceCacheSize = 16
-
+var execTraceCacheSize = 16
 var log = logging.Logger("statemgr")
 
 type StateManagerAPI interface {
@@ -74,6 +75,17 @@ func (m *migrationResultCache) keyForMigration(root cid.Cid) dstore.Key {
 	return dstore.NewKey(kStr)
 }
 
+func init() {
+	if s := os.Getenv("LOTUS_EXEC_TRACE_CACHE_SIZE"); s != "" {
+		letc, err := strconv.Atoi(s)
+		if err != nil {
+			log.Errorf("failed to parse 'LOTUS_EXEC_TRACE_CACHE_SIZE' env var: %s", err)
+		} else {
+			execTraceCacheSize = letc
+		}
+	}
+}
+
 func (m *migrationResultCache) Get(ctx context.Context, root cid.Cid) (cid.Cid, bool, error) {
 	k := m.keyForMigration(root)
 
@@ -200,9 +212,14 @@ func NewStateManager(cs *store.ChainStore, exec Executor, sys vm.SyscallBuilder,
 		}
 	}
 
-	execTraceCache, err := lru.NewARC[types.TipSetKey, tipSetCacheEntry](execTraceCacheSize)
-	if err != nil {
-		return nil, err
+	log.Debugf("execTraceCache size: %d", execTraceCacheSize)
+	var execTraceCache *lru.ARCCache[types.TipSetKey, tipSetCacheEntry]
+	var err error
+	if execTraceCacheSize > 0 {
+		execTraceCache, err = lru.NewARC[types.TipSetKey, tipSetCacheEntry](execTraceCacheSize)
+		if err != nil {
+			return nil, err
+		}
 	}
 
 	return &StateManager{

chain/stmgr/utils.go

@@ -72,7 +72,7 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
 
 	base, trace, err := sm.ExecutionTrace(ctx, ts)
 	if err != nil {
-		return cid.Undef, nil, err
+		return cid.Undef, nil, xerrors.Errorf("failed to compute base state: %w", err)
 	}
 
 	for i := ts.Height(); i < height; i++ {
@@ -116,6 +116,21 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
 		if ret.ExitCode != 0 {
 			log.Infof("compute state apply message %d failed (exit: %d): %s", i, ret.ExitCode, ret.ActorErr)
 		}
+
+		ir := &api.InvocResult{
+			MsgCid:         msg.Cid(),
+			Msg:            msg,
+			MsgRct:         &ret.MessageReceipt,
+			ExecutionTrace: ret.ExecutionTrace,
+			Duration:       ret.Duration,
+		}
+		if ret.ActorErr != nil {
+			ir.Error = ret.ActorErr.Error()
+		}
+		if ret.GasCosts != nil {
+			ir.GasCost = MakeMsgGasCost(msg, ret)
+		}
+		trace = append(trace, ir)
 	}
 
 	root, err := vmi.Flush(ctx)

chain/store/store.go

@@ -425,6 +425,11 @@ func (cs *ChainStore) MaybeTakeHeavierTipSet(ctx context.Context, ts *types.TipS
 	}
 
 	defer cs.heaviestLk.Unlock()
+
+	if ts.Equals(cs.heaviest) {
+		return nil
+	}
+
 	w, err := cs.weight(ctx, cs.StateBlockstore(), ts)
 	if err != nil {
 		return err

chain/sub/incoming.go

@@ -12,7 +12,7 @@ import (
 	bserv "github.com/ipfs/go-blockservice"
 	"github.com/ipfs/go-cid"
 	logging "github.com/ipfs/go-log/v2"
-	"github.com/ipni/storetheindex/announce/message"
+	"github.com/ipni/go-libipni/announce/message"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	"github.com/libp2p/go-libp2p/core/connmgr"
 	"github.com/libp2p/go-libp2p/core/peer"
@@ -358,7 +358,11 @@ func (mv *MessageValidator) Validate(ctx context.Context, pid peer.ID, msg *pubs
 			fallthrough
 		case xerrors.Is(err, messagepool.ErrNonceGap):
 			fallthrough
+		case xerrors.Is(err, messagepool.ErrGasFeeCapTooLow):
+			fallthrough
 		case xerrors.Is(err, messagepool.ErrNonceTooLow):
+			fallthrough
+		case xerrors.Is(err, messagepool.ErrExistingNonce):
 			return pubsub.ValidationIgnore
 		default:
 			return pubsub.ValidationReject

chain/sub/incoming_test.go

@@ -9,7 +9,7 @@ import (
 	"github.com/golang/mock/gomock"
 	blocks "github.com/ipfs/go-block-format"
 	"github.com/ipfs/go-cid"
-	"github.com/ipni/storetheindex/announce/message"
+	"github.com/ipni/go-libipni/announce/message"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	pb "github.com/libp2p/go-libp2p-pubsub/pb"
 	"github.com/libp2p/go-libp2p/core/peer"

chain/types/blockmsg.go

@@ -2,6 +2,7 @@ package types
 
 import (
 	"bytes"
+	"fmt"
 
 	"github.com/ipfs/go-cid"
 )
@@ -14,10 +15,13 @@ type BlockMsg struct {
 
 func DecodeBlockMsg(b []byte) (*BlockMsg, error) {
 	var bm BlockMsg
-	if err := bm.UnmarshalCBOR(bytes.NewReader(b)); err != nil {
+	data := bytes.NewReader(b)
+	if err := bm.UnmarshalCBOR(data); err != nil {
 		return nil, err
 	}
-
+	if l := data.Len(); l != 0 {
+		return nil, fmt.Errorf("extraneous data in BlockMsg CBOR encoding: got %d unexpected bytes", l)
+	}
 	return &bm, nil
 }
 

chain/types/blockmsg_test.go

@@ -0,0 +1,40 @@
+package types
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestDecodeBlockMsg(t *testing.T) {
+	type args struct {
+		b []byte
+	}
+	tests := []struct {
+		name    string
+		data    []byte
+		want    *BlockMsg
+		wantErr bool
+	}{
+		{"decode empty BlockMsg with extra data at the end", []byte{0x83, 0xf6, 0x80, 0x80, 0x20}, nil, true},
+		{"decode valid empty BlockMsg", []byte{0x83, 0xf6, 0x80, 0x80}, new(BlockMsg), false},
+		{"decode invalid cbor", []byte{0x83, 0xf6, 0x80}, nil, true},
+	}
+	for _, tt := range tests {
+		data := tt.data
+		want := tt.want
+		wantErr := tt.wantErr
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := DecodeBlockMsg(data)
+			if wantErr {
+				assert.Errorf(t, err, "DecodeBlockMsg(%x)", data)
+				return
+			}
+			assert.NoErrorf(t, err, "DecodeBlockMsg(%x)", data)
+			assert.Equalf(t, want, got, "DecodeBlockMsg(%x)", data)
+			serialized, err := got.Serialize()
+			assert.NoErrorf(t, err, "DecodeBlockMsg(%x)", data)
+			assert.Equalf(t, serialized, data, "DecodeBlockMsg(%x)", data)
+		})
+	}
+}

chain/types/ethtypes/eth_types.go

@@ -238,6 +238,30 @@ func (c *EthCall) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+type EthSyncingResult struct {
+	DoneSync      bool
+	StartingBlock EthUint64
+	CurrentBlock  EthUint64
+	HighestBlock  EthUint64
+}
+
+func (sr EthSyncingResult) MarshalJSON() ([]byte, error) {
+	if sr.DoneSync {
+		// when done syncing, the json response should be '"result": false'
+		return []byte("false"), nil
+	}
+
+	// need to do an anonymous struct to avoid infinite recursion
+	return json.Marshal(&struct {
+		StartingBlock EthUint64 `json:"startingblock"`
+		CurrentBlock  EthUint64 `json:"currentblock"`
+		HighestBlock  EthUint64 `json:"highestblock"`
+	}{
+		StartingBlock: sr.StartingBlock,
+		CurrentBlock:  sr.CurrentBlock,
+		HighestBlock:  sr.HighestBlock})
+}
+
 const (
 	EthAddressLength = 20
 	EthHashLength    = 32
@@ -548,12 +572,12 @@ func (h EthSubscriptionID) String() string {
 }
 
 type EthFilterSpec struct {
-	// Interpreted as an epoch or one of "latest" for last mined block, "earliest" for first,
+	// Interpreted as an epoch (in hex) or one of "latest" for last mined block, "earliest" for first,
 	// "pending" for not yet committed messages.
 	// Optional, default: "latest".
 	FromBlock *string `json:"fromBlock,omitempty"`
 
-	// Interpreted as an epoch or one of "latest" for last mined block, "earliest" for first,
+	// Interpreted as an epoch (in hex) or one of "latest" for last mined block, "earliest" for first,
 	// "pending" for not yet committed messages.
 	// Optional, default: "latest".
 	ToBlock *string `json:"toBlock,omitempty"`

cli/info.go

@@ -20,6 +20,7 @@ import (
 	"github.com/filecoin-project/lotus/api/v1api"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/journal/alerting"
 )
 
 var infoCmd = &cli.Command{
@@ -62,6 +63,21 @@ func infoCmdAct(cctx *cli.Context) error {
 	fmt.Printf(" [epoch %s]\n", color.MagentaString(("%d"), status.SyncStatus.Epoch))
 	fmt.Printf("Peers to: [publish messages %d] [publish blocks %d]\n", status.PeerStatus.PeersToPublishMsgs, status.PeerStatus.PeersToPublishBlocks)
 
+	alerts, err := fullapi.LogAlerts(ctx)
+	if err != nil {
+		fmt.Printf("ERROR: getting alerts: %s\n", err)
+	}
+
+	activeAlerts := make([]alerting.Alert, 0)
+	for _, alert := range alerts {
+		if alert.Active {
+			activeAlerts = append(activeAlerts, alert)
+		}
+	}
+	if len(activeAlerts) > 0 {
+		fmt.Printf("%s (check %s)\n", color.RedString("⚠ %d Active alerts", len(activeAlerts)), color.YellowString("lotus log alerts"))
+	}
+
 	//Chain health calculated as percentage: amount of blocks in last finality / very healthy amount of blocks in a finality (900 epochs * 5 blocks per tipset)
 	health := (100 * (900 * status.ChainStatus.BlocksPerTipsetLastFinality) / (900 * 5))
 	switch {

cli/state.go

@@ -1065,12 +1065,19 @@ var StateComputeStateCmd = &cli.Command{
 
 		ctx := ReqContext(cctx)
 
-		ts, err := LoadTipSet(ctx, cctx, api)
+		h := abi.ChainEpoch(cctx.Uint64("vm-height"))
+		var ts *types.TipSet
+		if tss := cctx.String("tipset"); tss != "" {
+			ts, err = ParseTipSetRef(ctx, api, tss)
+		} else if h > 0 {
+			ts, err = api.ChainGetTipSetByHeight(ctx, h, types.EmptyTSK)
+		} else {
+			ts, err = api.ChainHead(ctx)
+		}
 		if err != nil {
 			return err
 		}
 
-		h := abi.ChainEpoch(cctx.Uint64("vm-height"))
 		if h == 0 {
 			h = ts.Height()
 		}
@@ -1528,6 +1535,9 @@ func printMsg(ctx context.Context, api v0api.FullNode, msg cid.Cid, mw *lapi.Msg
 	if err := printReceiptReturn(ctx, api, m, mw.Receipt); err != nil {
 		return err
 	}
+	if mw.Receipt.EventsRoot != nil {
+		fmt.Printf("Events Root: %s\n", mw.Receipt.EventsRoot)
+	}
 
 	return nil
 }

cmd/lotus-bench/main.go

@@ -98,14 +98,16 @@ func main() {
 	log.Info("Starting lotus-bench")
 
 	app := &cli.App{
-		Name:    "lotus-bench",
-		Usage:   "Benchmark performance of lotus on your hardware",
-		Version: build.UserVersion(),
+		Name:                      "lotus-bench",
+		Usage:                     "Benchmark performance of lotus on your hardware",
+		Version:                   build.UserVersion(),
+		DisableSliceFlagSeparator: true,
 		Commands: []*cli.Command{
 			proveCmd,
 			sealBenchCmd,
 			simpleCmd,
 			importBenchCmd,
+			rpcCmd,
 		},
 	}
 

cmd/lotus-bench/rpc.go

@@ -0,0 +1,576 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"os/signal"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"text/tabwriter"
+	"time"
+
+	"github.com/urfave/cli/v2"
+)
+
+var rpcCmd = &cli.Command{
+	Name:  "rpc",
+	Usage: "Runs a concurrent stress test on one or more rpc methods and prints the performance metrics including latency distribution and histogram",
+	Description: `This benchmark is designed to stress test the rpc methods of a lotus node so that we can simulate real world usage and measure the performance of rpc methods on the node.
+	
+This benchmark has the following features:
+* Can query each method both sequentially and concurrently
+* Supports rate limiting
+* Can query multiple different endpoints at once (supporting different concurrency level and rate limiting for each method)
+* Gives a nice reporting summary of the stress testing of each method (including latency distribution, histogram and more)
+* Easy to use
+
+To use this benchmark you must specify the rpc methods you want to test using the --method options, the format of it is:
+
+  --method=NAME[:CONCURRENCY][:QPS][:PARAMS] where only NAME is required.
+
+Here are some real examples:
+  lotus-bench rpc --method='eth_chainId' // run eth_chainId with default concurrency and qps
+  lotus-bench rpc --method='eth_chainId:3'  // override concurrency to 3
+  lotus-bench rpc --method='eth_chainId::100' // override to 100 qps while using default concurrency
+  lotus-bench rpc --method='eth_chainId:3:100' // run using 3 workers but limit to 100 qps
+  lotus-bench rpc --method='eth_getTransactionCount:::["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run using optional params while using default concurrency and qps
+  lotus-bench rpc --method='eth_chainId' --method='eth_getTransactionCount:10:0:["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run multiple methods at once
+
+NOTE: The last two examples will not work until we upgrade urfave dependency (tracked in https://github.com/urfave/cli/issues/1734)`,
+	Flags: []cli.Flag{
+		&cli.StringFlag{
+			Name:  "endpoint",
+			Value: "http://127.0.0.1:1234/rpc/v1",
+			Usage: "The rpc endpoint to benchmark",
+		},
+		&cli.DurationFlag{
+			Name:  "duration",
+			Value: 60 * time.Second,
+			Usage: "Duration of benchmark in seconds",
+		},
+		&cli.IntFlag{
+			Name:  "concurrency",
+			Value: 10,
+			Usage: "How many workers should be used per rpc method (can be overridden per method)",
+		},
+		&cli.IntFlag{
+			Name:  "qps",
+			Value: 0,
+			Usage: "How many requests per second should be sent per rpc method (can be overridden per method), a value of 0 means no limit",
+		},
+		&cli.StringSliceFlag{
+			Name: "method",
+			Usage: `Method to benchmark, you can specify multiple methods by repeating this flag. You can also specify method specific options to set the concurrency and qps for each method (see usage).
+`,
+		},
+		&cli.DurationFlag{
+			Name:  "watch",
+			Value: 0 * time.Second,
+			Usage: "If >0 then generates reports every N seconds (only supports linux/unix)",
+		},
+		&cli.BoolFlag{
+			Name:  "print-response",
+			Value: false,
+			Usage: "print the response of each request",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		if len(cctx.StringSlice("method")) == 0 {
+			return errors.New("you must specify and least one method to benchmark")
+		}
+
+		var rpcMethods []*RPCMethod
+		for _, str := range cctx.StringSlice("method") {
+			entries := strings.SplitN(str, ":", 4)
+			if len(entries) == 0 {
+				return errors.New("invalid method format")
+			}
+
+			// check if concurrency was specified
+			concurrency := cctx.Int("concurrency")
+			if len(entries) > 1 {
+				if len(entries[1]) > 0 {
+					var err error
+					concurrency, err = strconv.Atoi(entries[1])
+					if err != nil {
+						return fmt.Errorf("could not parse concurrency value from method %s: %v", entries[0], err)
+					}
+				}
+			}
+
+			// check if qps was specified
+			qps := cctx.Int("qps")
+			if len(entries) > 2 {
+				if len(entries[2]) > 0 {
+					var err error
+					qps, err = strconv.Atoi(entries[2])
+					if err != nil {
+						return fmt.Errorf("could not parse qps value from method %s: %v", entries[0], err)
+					}
+				}
+			}
+
+			// check if params was specified
+			params := "[]"
+			if len(entries) > 3 {
+				params = entries[3]
+			}
+
+			rpcMethods = append(rpcMethods, &RPCMethod{
+				w:           os.Stdout,
+				uri:         cctx.String("endpoint"),
+				method:      entries[0],
+				concurrency: concurrency,
+				qps:         qps,
+				params:      params,
+				printResp:   cctx.Bool("print-response"),
+			})
+		}
+
+		// terminate early on ctrl+c
+		c := make(chan os.Signal, 1)
+		signal.Notify(c, os.Interrupt)
+		go func() {
+			<-c
+			fmt.Println("Received interrupt, stopping...")
+			for _, method := range rpcMethods {
+				method.Stop()
+			}
+		}()
+
+		// stop all threads after duration
+		go func() {
+			time.Sleep(cctx.Duration("duration"))
+			for _, e := range rpcMethods {
+				e.Stop()
+			}
+		}()
+
+		// start all threads
+		var wg sync.WaitGroup
+		wg.Add(len(rpcMethods))
+
+		for _, e := range rpcMethods {
+			go func(e *RPCMethod) {
+				defer wg.Done()
+				err := e.Run()
+				if err != nil {
+					fmt.Printf("error running rpc method: %v\n", err)
+				}
+			}(e)
+		}
+
+		// if watch is set then print a report every N seconds
+		var progressCh chan struct{}
+		if cctx.Duration("watch") > 0 {
+			progressCh = make(chan struct{}, 1)
+			go func(progressCh chan struct{}) {
+				ticker := time.NewTicker(cctx.Duration("watch"))
+				for {
+					clearAndPrintReport := func() {
+						// clear the screen move the curser to the top left
+						fmt.Print("\033[2J")
+						fmt.Printf("\033[%d;%dH", 1, 1)
+						for i, e := range rpcMethods {
+							e.Report()
+							if i < len(rpcMethods)-1 {
+								fmt.Println()
+							}
+						}
+					}
+					select {
+					case <-ticker.C:
+						clearAndPrintReport()
+					case <-progressCh:
+						clearAndPrintReport()
+						return
+					}
+				}
+			}(progressCh)
+		}
+
+		wg.Wait()
+
+		if progressCh != nil {
+			// wait for the watch go routine to return
+			progressCh <- struct{}{}
+
+			// no need to print the report again
+			return nil
+		}
+
+		// print the report for each endpoint
+		for i, e := range rpcMethods {
+			e.Report()
+			if i < len(rpcMethods)-1 {
+				fmt.Println()
+			}
+		}
+
+		return nil
+	},
+}
+
+// RPCMethod handles the benchmarking of a single endpoint method.
+type RPCMethod struct {
+	w io.Writer
+	// the endpoint uri
+	uri string
+	// the rpc method we want to benchmark
+	method string
+	// the number of concurrent requests to make to this endpoint
+	concurrency int
+	// if >0 then limit to qps is the max number of requests per second to make to this endpoint (0 = no limit)
+	qps int
+	// many endpoints require specific parameters to be passed
+	params string
+	// whether or not to print the response of each request (useful for debugging)
+	printResp bool
+	// instruct the worker go routines to stop
+	stopCh chan struct{}
+	// when the endpoint bencharking started
+	start time.Time
+	// results channel is used by the workers to send results to the reporter
+	results chan *result
+	// reporter handles reading the results from workers and printing the report statistics
+	reporter *Reporter
+}
+
+// result is the result of a single rpc method request.
+type result struct {
+	err        error
+	statusCode *int
+	duration   time.Duration
+}
+
+func (rpc *RPCMethod) Run() error {
+	client := &http.Client{
+		Timeout: 0,
+	}
+
+	var wg sync.WaitGroup
+	wg.Add(rpc.concurrency)
+
+	rpc.results = make(chan *result, rpc.concurrency*1_000)
+	rpc.stopCh = make(chan struct{}, rpc.concurrency)
+
+	go func() {
+		rpc.reporter = NewReporter(rpc.results, rpc.w)
+		rpc.reporter.Run()
+	}()
+
+	rpc.start = time.Now()
+
+	// throttle the number of requests per second
+	var qpsTicker *time.Ticker
+	if rpc.qps > 0 {
+		qpsTicker = time.NewTicker(time.Second / time.Duration(rpc.qps))
+	}
+
+	for i := 0; i < rpc.concurrency; i++ {
+		go func() {
+			rpc.startWorker(client, qpsTicker)
+			wg.Done()
+		}()
+	}
+	wg.Wait()
+
+	// close the results channel so reporter will stop
+	close(rpc.results)
+
+	// wait until the reporter is done
+	<-rpc.reporter.doneCh
+
+	return nil
+}
+
+func (rpc *RPCMethod) startWorker(client *http.Client, qpsTicker *time.Ticker) {
+	for {
+		// check if we should stop
+		select {
+		case <-rpc.stopCh:
+			return
+		default:
+		}
+
+		// wait for the next tick if we are rate limiting this endpoint
+		if qpsTicker != nil {
+			<-qpsTicker.C
+		}
+
+		req, err := rpc.buildRequest()
+		if err != nil {
+			log.Fatalln(err)
+		}
+
+		start := time.Now()
+
+		var statusCode *int
+
+		// send request the endpoint
+		resp, err := client.Do(req)
+		if err != nil {
+			err = fmt.Errorf("HTTP error: %s", err.Error())
+		} else {
+			statusCode = &resp.StatusCode
+
+			// there was not a HTTP error but we need to still check the json response for errrors
+			var data []byte
+			data, err = io.ReadAll(resp.Body)
+			if err != nil {
+				log.Fatalln(err)
+			}
+
+			// we are only interested if it has the error field in the response
+			type respData struct {
+				Error struct {
+					Code    int    `json:"code"`
+					Message string `json:"message"`
+				} `json:"error"`
+			}
+
+			// unmarshal the response into a struct so we can check for errors
+			var d respData
+			err = json.Unmarshal(data, &d)
+			if err != nil {
+				log.Fatalln(err)
+			}
+
+			// if the response has an error json message then it should be considered an error just like any http error
+			if len(d.Error.Message) > 0 {
+				// truncate the error message if it is too long
+				if len(d.Error.Message) > 1000 {
+					d.Error.Message = d.Error.Message[:1000] + "..."
+				}
+				// remove newlines from the error message so we don't screw up the report
+				d.Error.Message = strings.ReplaceAll(d.Error.Message, "\n", "")
+
+				err = fmt.Errorf("JSON error: code:%d, message:%s", d.Error.Code, d.Error.Message)
+			}
+
+			if rpc.printResp {
+				fmt.Printf("[%s] %s", rpc.method, string(data))
+			}
+
+			resp.Body.Close() //nolint:errcheck
+		}
+
+		rpc.results <- &result{
+			statusCode: statusCode,
+			err:        err,
+			duration:   time.Since(start),
+		}
+	}
+}
+
+func (rpc *RPCMethod) buildRequest() (*http.Request, error) {
+	jreq, err := json.Marshal(struct {
+		Jsonrpc string          `json:"jsonrpc"`
+		ID      int             `json:"id"`
+		Method  string          `json:"method"`
+		Params  json.RawMessage `json:"params"`
+	}{
+		Jsonrpc: "2.0",
+		Method:  rpc.method,
+		Params:  json.RawMessage(rpc.params),
+		ID:      0,
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	req, err := http.NewRequest("POST", rpc.uri, bytes.NewReader(jreq))
+	if err != nil {
+		return nil, err
+	}
+
+	req.Header.Set("Accept", "application/json")
+
+	return req, nil
+}
+
+func (rpc *RPCMethod) Stop() {
+	for i := 0; i < rpc.concurrency; i++ {
+		rpc.stopCh <- struct{}{}
+	}
+}
+
+func (rpc *RPCMethod) Report() {
+	total := time.Since(rpc.start)
+	fmt.Fprintf(rpc.w, "[%s]:\n", rpc.method)
+	fmt.Fprintf(rpc.w, "- Options:\n")
+	fmt.Fprintf(rpc.w, "  - concurrency: %d\n", rpc.concurrency)
+	fmt.Fprintf(rpc.w, "  - params: %s\n", rpc.params)
+	fmt.Fprintf(rpc.w, "  - qps: %d\n", rpc.qps)
+	rpc.reporter.Print(total, rpc.w)
+}
+
+// Reporter reads the results from the workers through the results channel and aggregates the results.
+type Reporter struct {
+	// write the report to this writer
+	w io.Writer
+	// the reporter read the results from this channel
+	results chan *result
+	// doneCh is used to signal that the reporter has finished reading the results (channel has closed)
+	doneCh chan bool
+
+	// lock protect the following fields during critical sections (if --watch was specified)
+	lock sync.Mutex
+	// the latencies of all requests
+	latencies []int64
+	// the number of requests that returned each status code
+	statusCodes map[int]int
+	// the number of errors that occurred
+	errors map[string]int
+}
+
+func NewReporter(results chan *result, w io.Writer) *Reporter {
+	return &Reporter{
+		w:           w,
+		results:     results,
+		doneCh:      make(chan bool, 1),
+		statusCodes: make(map[int]int),
+		errors:      make(map[string]int),
+	}
+}
+
+func (r *Reporter) Run() {
+	for res := range r.results {
+		r.lock.Lock()
+
+		r.latencies = append(r.latencies, res.duration.Milliseconds())
+
+		if res.statusCode != nil {
+			r.statusCodes[*res.statusCode]++
+		}
+
+		if res.err != nil {
+			if len(r.errors) < 1_000_000 {
+				r.errors[res.err.Error()]++
+			} else {
+				// we don't want to store too many errors in memory
+				r.errors["hidden"]++
+			}
+		} else {
+			r.errors["nil"]++
+		}
+
+		r.lock.Unlock()
+	}
+
+	r.doneCh <- true
+}
+
+func (r *Reporter) Print(elapsed time.Duration, w io.Writer) {
+	r.lock.Lock()
+	defer r.lock.Unlock()
+
+	nrReq := int64(len(r.latencies))
+	if nrReq == 0 {
+		fmt.Println("No requests were made")
+		return
+	}
+
+	// we need to sort the latencies slice to calculate the percentiles
+	sort.Slice(r.latencies, func(i, j int) bool {
+		return r.latencies[i] < r.latencies[j]
+	})
+
+	var totalLatency int64 = 0
+	for _, latency := range r.latencies {
+		totalLatency += latency
+	}
+
+	fmt.Fprintf(w, "- Total Requests: %d\n", nrReq)
+	fmt.Fprintf(w, "- Total Duration: %dms\n", elapsed.Milliseconds())
+	fmt.Fprintf(w, "- Requests/sec: %f\n", float64(nrReq)/elapsed.Seconds())
+	fmt.Fprintf(w, "- Avg latency: %dms\n", totalLatency/nrReq)
+	fmt.Fprintf(w, "- Median latency: %dms\n", r.latencies[nrReq/2])
+	fmt.Fprintf(w, "- Latency distribution:\n")
+	percentiles := []float64{0.1, 0.5, 0.9, 0.95, 0.99, 0.999}
+	for _, p := range percentiles {
+		idx := int64(p * float64(nrReq))
+		fmt.Fprintf(w, "    %s%% in %dms\n", fmt.Sprintf("%.2f", p*100.0), r.latencies[idx])
+	}
+
+	// create a simple histogram with 10 buckets spanning the range of latency
+	// into equal ranges
+	//
+	nrBucket := 10
+	buckets := make([]Bucket, nrBucket)
+	latencyRange := r.latencies[len(r.latencies)-1]
+	bucketRange := latencyRange / int64(nrBucket)
+
+	// mark the end of each bucket
+	for i := 0; i < nrBucket; i++ {
+		buckets[i].start = int64(i) * bucketRange
+		buckets[i].end = buckets[i].start + bucketRange
+		// extend the last bucked by any remaning range caused by the integer division
+		if i == nrBucket-1 {
+			buckets[i].end = latencyRange
+		}
+	}
+
+	// count the number of requests in each bucket
+	currBucket := 0
+	for i := 0; i < len(r.latencies); {
+		if r.latencies[i] <= buckets[currBucket].end {
+			buckets[currBucket].cnt++
+			i++
+		} else {
+			currBucket++
+		}
+	}
+
+	// print the histogram using a tabwriter which will align the columns nicely
+	fmt.Fprintf(w, "- Histogram:\n")
+	const padding = 2
+	tabWriter := tabwriter.NewWriter(w, 0, 0, padding, ' ', tabwriter.AlignRight|tabwriter.Debug)
+	for i := 0; i < nrBucket; i++ {
+		ratio := float64(buckets[i].cnt) / float64(nrReq)
+		bars := strings.Repeat("#", int(ratio*100))
+		fmt.Fprintf(tabWriter, "  %d-%dms\t%d\t%s (%s%%)\n", buckets[i].start, buckets[i].end, buckets[i].cnt, bars, fmt.Sprintf("%.2f", ratio*100))
+	}
+	tabWriter.Flush() //nolint:errcheck
+
+	fmt.Fprintf(w, "- Status codes:\n")
+	for code, cnt := range r.statusCodes {
+		fmt.Fprintf(w, "    [%d]: %d\n", code, cnt)
+	}
+
+	// print the 10 most occurring errors (in case error values are not unique)
+	//
+	type kv struct {
+		err string
+		cnt int
+	}
+	var sortedErrors []kv
+	for err, cnt := range r.errors {
+		sortedErrors = append(sortedErrors, kv{err, cnt})
+	}
+	sort.Slice(sortedErrors, func(i, j int) bool {
+		return sortedErrors[i].cnt > sortedErrors[j].cnt
+	})
+	fmt.Fprintf(w, "- Errors (top 10):\n")
+	for i, se := range sortedErrors {
+		if i > 10 {
+			break
+		}
+		fmt.Fprintf(w, "    [%s]: %d\n", se.err, se.cnt)
+	}
+}
+
+type Bucket struct {
+	start int64
+	// the end value of the bucket
+	end int64
+	// how many entries are in the bucket
+	cnt int
+}

cmd/lotus-fountain/main.go

@@ -7,6 +7,7 @@ import (
 	"net"
 	"net/http"
 	"os"
+	"strings"
 	"time"
 
 	rice "github.com/GeertJohan/go.rice"
@@ -15,10 +16,14 @@ import (
 	"golang.org/x/xerrors"
 
 	"github.com/filecoin-project/go-address"
+	verifregtypes9 "github.com/filecoin-project/go-state-types/builtin/v9/verifreg"
 
 	"github.com/filecoin-project/lotus/api/v0api"
 	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/chain/actors"
+	"github.com/filecoin-project/lotus/chain/actors/builtin/verifreg"
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
 	lcli "github.com/filecoin-project/lotus/cli"
 )
 
@@ -70,6 +75,11 @@ var runCmd = &cli.Command{
 			EnvVars: []string{"LOTUS_FOUNTAIN_AMOUNT"},
 			Value:   "50",
 		},
+		&cli.Uint64Flag{
+			Name:    "data-cap",
+			EnvVars: []string{"LOTUS_DATACAP_AMOUNT"},
+			Value:   verifregtypes9.MinVerifiedDealSize.Uint64(),
+		},
 		&cli.Float64Flag{
 			Name:  "captcha-threshold",
 			Value: 0.5,
@@ -108,6 +118,7 @@ var runCmd = &cli.Command{
 			ctx:            ctx,
 			api:            nodeApi,
 			from:           from,
+			allowance:      types.NewInt(cctx.Uint64("data-cap")),
 			sendPerRequest: sendPerRequest,
 			limiter: NewLimiter(LimiterConfig{
 				TotalRate:   500 * time.Millisecond,
@@ -124,6 +135,8 @@ var runCmd = &cli.Command{
 		http.Handle("/", http.FileServer(box.HTTPBox()))
 		http.HandleFunc("/funds.html", prepFundsHtml(box))
 		http.Handle("/send", h)
+		http.HandleFunc("/datacap.html", prepDataCapHtml(box))
+		http.Handle("/datacap", h)
 		fmt.Printf("Open http://%s\n", cctx.String("front"))
 
 		go func() {
@@ -156,12 +169,24 @@ func prepFundsHtml(box *rice.Box) http.HandlerFunc {
 	}
 }
 
+func prepDataCapHtml(box *rice.Box) http.HandlerFunc {
+	tmpl := template.Must(template.New("datacaps").Parse(box.MustString("datacap.html")))
+	return func(w http.ResponseWriter, r *http.Request) {
+		err := tmpl.Execute(w, os.Getenv("RECAPTCHA_SITE_KEY"))
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadGateway)
+			return
+		}
+	}
+}
+
 type handler struct {
 	ctx context.Context
 	api v0api.FullNode
 
 	from           address.Address
 	sendPerRequest types.FIL
+	allowance      types.BigInt
 
 	limiter        *Limiter
 	recapThreshold float64
@@ -187,24 +212,41 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		http.Error(w, err.Error(), http.StatusBadGateway)
 		return
 	}
+
 	if !capResp.Success || capResp.Score < h.recapThreshold {
 		log.Infow("spam", "capResp", capResp)
 		http.Error(w, "spam protection", http.StatusUnprocessableEntity)
 		return
 	}
 
-	to, err := address.NewFromString(r.FormValue("address"))
-	if err != nil {
+	addressInput := r.FormValue("address")
+
+	var filecoinAddress address.Address
+	var decodeError error
+
+	if strings.HasPrefix(addressInput, "0x") {
+		ethAddress, err := ethtypes.ParseEthAddress(addressInput)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+
+		filecoinAddress, decodeError = ethAddress.ToFilecoinAddress()
+	} else {
+		filecoinAddress, decodeError = address.NewFromString(addressInput)
+	}
+
+	if decodeError != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return
 	}
-	if to == address.Undef {
+	if filecoinAddress == address.Undef {
 		http.Error(w, "empty address", http.StatusBadRequest)
 		return
 	}
 
 	// Limit based on wallet address
-	limiter := h.limiter.GetWalletLimiter(to.String())
+	limiter := h.limiter.GetWalletLimiter(filecoinAddress.String())
 	if !limiter.Allow() {
 		http.Error(w, http.StatusText(http.StatusTooManyRequests)+": wallet limit", http.StatusTooManyRequests)
 		return
@@ -227,11 +269,37 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	smsg, err := h.api.MpoolPushMessage(h.ctx, &types.Message{
-		Value: types.BigInt(h.sendPerRequest),
-		From:  h.from,
-		To:    to,
-	}, nil)
+	var smsg *types.SignedMessage
+	if r.RequestURI == "/send" {
+		smsg, err = h.api.MpoolPushMessage(
+			h.ctx, &types.Message{
+				Value: types.BigInt(h.sendPerRequest),
+				From:  h.from,
+				To:    filecoinAddress,
+			}, nil)
+	} else if r.RequestURI == "/datacap" {
+		var params []byte
+		params, err = actors.SerializeParams(
+			&verifregtypes9.AddVerifiedClientParams{
+				Address:   filecoinAddress,
+				Allowance: h.allowance,
+			})
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		smsg, err = h.api.MpoolPushMessage(
+			h.ctx, &types.Message{
+				Params: params,
+				From:   h.from,
+				To:     verifreg.Address,
+				Method: verifreg.Methods.AddVerifiedClient,
+			}, nil)
+	} else {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
 	if err != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return

cmd/lotus-fountain/site/datacap.html

@@ -0,0 +1,41 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Grant DataCap - Lotus Fountain</title>
+    <link rel="stylesheet" type="text/css" href="main.css">
+	<script src="https://www.google.com/recaptcha/api.js"></script>
+	<script>
+   		function onSubmit(token) {
+     	document.getElementById("datacap-form").submit();
+   	}
+	</script>
+
+</head>
+<body>
+<div class="Index">
+    <div class="Index-nodes">
+        <div class="Index-node">
+            <h3>Grant datacap</h3>
+            <p>Please input your address to receive a data cap on the Calibration Testnet.</p>
+        </div>
+        <div class="Index-node">
+            <form action='/datacap' method='post' id='datacap-form'>
+                <span>Enter destination address:</span>
+                <input type='text' name='address' style="width: 300px" placeholder="t0/1/2/3/4 or 0xETH">
+                <button class="g-recaptcha"
+                        data-sitekey="{{ . }}"
+                        data-callback='onSubmit'
+                        data-action='submit'>Grant Datacap</button>
+            </form>
+        </div>
+    </div>
+
+    <div class="Index-footer">
+        <div>
+            <a href="index.html">[Back]</a>
+            <span style="float: right">Not dispensing real Filecoin tokens</span>
+        </div>
+    </div>
+</div>
+</body>
+</html>

cmd/lotus-fountain/site/funds.html

@@ -15,12 +15,13 @@
 <div class="Index">
     <div class="Index-nodes">
         <div class="Index-node">
-            [SENDING FUNDS]
+            <h3>Send funds</h3>
+            <p>Please input your address to receive test FIL (tFIL) on the Calibration Testnet. This faucet dispenses 100 tFIL.</p>
         </div>
         <div class="Index-node">
             <form action='/send' method='post' id='funds-form'>
                 <span>Enter destination address:</span>
-				<input type='text' name='address' style="width: 300px">
+                <input type='text' name='address' style="width: 300px" placeholder="Enter t0/1/2/3/4 or 0xETH">
 				<button class="g-recaptcha" 
 						data-sitekey="{{ . }}"
 						data-callback='onSubmit' 

cmd/lotus-fountain/site/index.html

@@ -8,10 +8,16 @@
 <div class="Index">
     <div class="Index-nodes">
         <div class="Index-node">
-            [LOTUS DEVNET FAUCET]
+            LOTUS DEVNET FAUCET
         </div>
         <div class="Index-node">
-            <a href="funds.html">[Send Funds]</a>
+            <a href="funds.html">Send Funds</a>
+        </div>
+        <div class="Index-node">
+            LOTUS DEVNET GRANT DATACAP
+        </div>
+        <div class="Index-node">
+            <a href="datacap.html">Grant DataCap</a>
         </div>
     </div>
     <div class="Index-footer">

cmd/lotus-fountain/site/main.css

@@ -1,7 +1,6 @@
 body {
-    font-family: 'monospace';
-    background: #1f1f1f;
-    color: #f0f0f0;
+    font-family: 'Helvetica Neue', sans-serif;
+    background-color: #f0f0f0;
     padding: 0;
     margin: 0;
 }
@@ -9,21 +8,22 @@ body {
 .Index {
     width: 100vw;
     height: 100vh;
-    background: #1a1a1a;
-    color: #f0f0f0;
-    font-family: monospace;
+    background-color: #f0f0f0;
+    color: #333;
+    font-family: 'Helvetica Neue', sans-serif;
 
     display: grid;
     grid-template-columns: auto 40vw auto;
     grid-template-rows: auto auto auto 3em;
     grid-template-areas:
-            ". . . ."
-            ". main main ."
-            ". . . ."
-            "footer footer footer footer";
+    ". . . ."
+    ". main main ."
+    ". . . ."
+    "footer footer footer footer";
 }
+
 .Index-footer {
-    background: #2a2a2a;
+    background-color: #333;
     grid-area: footer;
 }
 
@@ -34,23 +34,49 @@ body {
 
 .Index-nodes {
     grid-area: main;
-    background: #2a2a2a;
 }
 
 .Index-node {
     margin: 5px;
     padding: 15px;
-    background: #1f1f1f;
+    background-color: #fff;
+    box-shadow: 0 0 5px rgba(0, 0, 0, 0.1);
+}
+
+span {
+    display: block;
+    margin-bottom: 5px;
+}
+
+input[type="text"] {
+    width: 100%;
+    padding: 10px;
+    border-radius: 5px;
+    border: 1px solid #ccc;
+    margin-bottom: 10px;
+}
+
+button {
+    background-color: #4c9aff;
+    color: #fff;
+    border: none;
+    border-radius: 5px;
+    padding: 10px 20px;
+    font-size: 1.2em;
+}
+
+button:hover {
+    background-color: #4c7eff;
 }
 
 a:link {
-    color: #50f020;
+    color: #333;
 }
 
 a:visited {
-    color: #50f020;
+    color: #333;
 }
 
 a:hover {
-    color: #30a00a;
+    color: #555;
 }

cmd/lotus-miner/actor.go

@@ -1166,7 +1166,7 @@ var actorConfirmChangeWorker = &cli.Command{
 var actorConfirmChangeBeneficiary = &cli.Command{
 	Name:      "confirm-change-beneficiary",
 	Usage:     "Confirm a beneficiary address change",
-	ArgsUsage: "[minerAddress]",
+	ArgsUsage: "[minerID]",
 	Flags: []cli.Flag{
 		&cli.BoolFlag{
 			Name:  "really-do-it",

cmd/lotus-miner/sectors.go

@@ -2,9 +2,11 @@ package main
 
 import (
 	"bufio"
+	"encoding/csv"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"math"
 	"os"
 	"sort"
 	"strconv"
@@ -318,6 +320,9 @@ var sectorsListCmd = &cli.Command{
 			Value: parallelSectorChecks,
 		},
 	},
+	Subcommands: []*cli.Command{
+		sectorsListUpgradeBoundsCmd,
+	},
 	Action: func(cctx *cli.Context) error {
 		// http mode allows for parallel json decoding/encoding, which was a bottleneck here
 		minerApi, closer, err := lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
@@ -585,6 +590,169 @@ var sectorsListCmd = &cli.Command{
 	},
 }
 
+var sectorsListUpgradeBoundsCmd = &cli.Command{
+	Name:  "upgrade-bounds",
+	Usage: "Output upgrade bounds for available sectors",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "buckets",
+			Value: 25,
+		},
+		&cli.BoolFlag{
+			Name:  "csv",
+			Usage: "output machine-readable values",
+		},
+		&cli.BoolFlag{
+			Name:  "deal-terms",
+			Usage: "bucket by how many deal-sectors can start at a given expiration",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		minerApi, closer, err := lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		fullApi, closer2, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer2()
+
+		ctx := lcli.ReqContext(cctx)
+
+		list, err := minerApi.SectorsListInStates(ctx, []api.SectorState{
+			api.SectorState(sealing.Available),
+		})
+		if err != nil {
+			return xerrors.Errorf("getting sector list: %w", err)
+		}
+
+		head, err := fullApi.ChainHead(ctx)
+		if err != nil {
+			return xerrors.Errorf("getting chain head: %w", err)
+		}
+
+		filter := bitfield.New()
+
+		for _, s := range list {
+			filter.Set(uint64(s))
+		}
+
+		maddr, err := minerApi.ActorAddress(ctx)
+		if err != nil {
+			return err
+		}
+
+		sset, err := fullApi.StateMinerSectors(ctx, maddr, &filter, head.Key())
+		if err != nil {
+			return err
+		}
+
+		if len(sset) == 0 {
+			return nil
+		}
+
+		var minExpiration, maxExpiration abi.ChainEpoch
+
+		for _, s := range sset {
+			if s.Expiration < minExpiration || minExpiration == 0 {
+				minExpiration = s.Expiration
+			}
+			if s.Expiration > maxExpiration {
+				maxExpiration = s.Expiration
+			}
+		}
+
+		buckets := cctx.Int("buckets")
+		bucketSize := (maxExpiration - minExpiration) / abi.ChainEpoch(buckets)
+		bucketCounts := make([]int, buckets+1)
+
+		for b := range bucketCounts {
+			bucketMin := minExpiration + abi.ChainEpoch(b)*bucketSize
+			bucketMax := minExpiration + abi.ChainEpoch(b+1)*bucketSize
+
+			if cctx.Bool("deal-terms") {
+				bucketMax = bucketMax + policy.MarketDefaultAllocationTermBuffer
+			}
+
+			for _, s := range sset {
+				isInBucket := s.Expiration >= bucketMin && s.Expiration < bucketMax
+
+				if isInBucket {
+					bucketCounts[b]++
+				}
+			}
+
+		}
+
+		// Creating CSV writer
+		writer := csv.NewWriter(os.Stdout)
+
+		// Writing CSV headers
+		err = writer.Write([]string{"Max Expiration in Bucket", "Sector Count"})
+		if err != nil {
+			return xerrors.Errorf("writing csv headers: %w", err)
+		}
+
+		// Writing bucket details
+
+		if cctx.Bool("csv") {
+			for i := 0; i < buckets; i++ {
+				maxExp := minExpiration + abi.ChainEpoch(i+1)*bucketSize
+
+				timeStr := strconv.FormatInt(int64(maxExp), 10)
+
+				err = writer.Write([]string{
+					timeStr,
+					strconv.Itoa(bucketCounts[i]),
+				})
+				if err != nil {
+					return xerrors.Errorf("writing csv row: %w", err)
+				}
+			}
+
+			// Flush to make sure all data is written to the underlying writer
+			writer.Flush()
+
+			if err := writer.Error(); err != nil {
+				return xerrors.Errorf("flushing csv writer: %w", err)
+			}
+
+			return nil
+		}
+
+		tw := tablewriter.New(
+			tablewriter.Col("Bucket Expiration"),
+			tablewriter.Col("Sector Count"),
+			tablewriter.Col("Bar"),
+		)
+
+		var barCols = 40
+		var maxCount int
+
+		for _, c := range bucketCounts {
+			if c > maxCount {
+				maxCount = c
+			}
+		}
+
+		for i := 0; i < buckets; i++ {
+			maxExp := minExpiration + abi.ChainEpoch(i+1)*bucketSize
+			timeStr := cliutil.EpochTime(head.Height(), maxExp)
+
+			tw.Write(map[string]interface{}{
+				"Bucket Expiration": timeStr,
+				"Sector Count":      color.YellowString("%d", bucketCounts[i]),
+				"Bar":               "[" + color.GreenString(strings.Repeat("|", bucketCounts[i]*barCols/maxCount)) + strings.Repeat(" ", barCols-bucketCounts[i]*barCols/maxCount) + "]",
+			})
+		}
+
+		return tw.Flush(os.Stdout)
+	},
+}
+
 var sectorsRefsCmd = &cli.Command{
 	Name:  "refs",
 	Usage: "List References to sectors",
@@ -1922,10 +2090,31 @@ var sectorsBatchingPendingCommit = &cli.Command{
 			for _, sector := range pending {
 				fmt.Println(sector.Number)
 			}
-			return nil
-		}
 
-		fmt.Println("No sectors queued to be committed")
+			reader := bufio.NewReader(os.Stdin)
+			fmt.Print("Do you want to publish these sectors now? (yes/no): ")
+			userInput, err := reader.ReadString('\n')
+			if err != nil {
+				return xerrors.Errorf("reading user input: %w", err)
+			}
+			userInput = strings.ToLower(strings.TrimSpace(userInput))
+
+			if userInput == "yes" {
+				err := cctx.Set("publish-now", "true")
+				if err != nil {
+					return xerrors.Errorf("setting publish-now flag: %w", err)
+				}
+				return cctx.Command.Action(cctx)
+			} else if userInput == "no" {
+				return nil
+			} else {
+				fmt.Println("Invalid input. Please answer with 'yes' or 'no'.")
+				return nil
+			}
+
+		} else {
+			fmt.Println("No sectors queued to be committed")
+		}
 		return nil
 	},
 }
@@ -1980,10 +2169,31 @@ var sectorsBatchingPendingPreCommit = &cli.Command{
 			for _, sector := range pending {
 				fmt.Println(sector.Number)
 			}
-			return nil
-		}
 
-		fmt.Println("No sectors queued to be committed")
+			reader := bufio.NewReader(os.Stdin)
+			fmt.Print("Do you want to publish these sectors now? (yes/no): ")
+			userInput, err := reader.ReadString('\n')
+			if err != nil {
+				return xerrors.Errorf("reading user input: %w", err)
+			}
+			userInput = strings.ToLower(strings.TrimSpace(userInput))
+
+			if userInput == "yes" {
+				err := cctx.Set("publish-now", "true")
+				if err != nil {
+					return xerrors.Errorf("setting publish-now flag: %w", err)
+				}
+				return cctx.Command.Action(cctx)
+			} else if userInput == "no" {
+				return nil
+			} else {
+				fmt.Println("Invalid input. Please answer with 'yes' or 'no'.")
+				return nil
+			}
+
+		} else {
+			fmt.Println("No sectors queued to be committed")
+		}
 		return nil
 	},
 }
@@ -2014,7 +2224,6 @@ func yesno(b bool) string {
 	return color.RedString("NO")
 }
 
-// TODO simulate this call if --really-do-it is not used
 var sectorsCompactPartitionsCmd = &cli.Command{
 	Name:  "compact-partitions",
 	Usage: "removes dead sectors from partitions and reduces the number of partitions used if possible",
@@ -2040,12 +2249,7 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		if !cctx.Bool("really-do-it") {
-			fmt.Println("Pass --really-do-it to actually execute this action")
-			return nil
-		}
-
-		api, acloser, err := lcli.GetFullNodeAPI(cctx)
+		fullNodeAPI, acloser, err := lcli.GetFullNodeAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -2058,7 +2262,7 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 			return err
 		}
 
-		minfo, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+		minfo, err := fullNodeAPI.StateMinerInfo(ctx, maddr, types.EmptyTSK)
 		if err != nil {
 			return err
 		}
@@ -2074,46 +2278,118 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 		}
 		fmt.Printf("compacting %d paritions\n", len(parts))
 
+		var makeMsgForPartitions func(partitionsBf bitfield.BitField) ([]*types.Message, error)
+		makeMsgForPartitions = func(partitionsBf bitfield.BitField) ([]*types.Message, error) {
+			params := miner.CompactPartitionsParams{
+				Deadline:   deadline,
+				Partitions: partitionsBf,
+			}
+
+			sp, aerr := actors.SerializeParams(&params)
+			if aerr != nil {
+				return nil, xerrors.Errorf("serializing params: %w", err)
+			}
+
+			msg := &types.Message{
+				From:   minfo.Worker,
+				To:     maddr,
+				Method: builtin.MethodsMiner.CompactPartitions,
+				Value:  big.Zero(),
+				Params: sp,
+			}
+
+			estimatedMsg, err := fullNodeAPI.GasEstimateMessageGas(ctx, msg, nil, types.EmptyTSK)
+			if err != nil && xerrors.Is(err, &api.ErrOutOfGas{}) {
+				// the message is too big -- split into 2
+				partitionsSlice, err := partitionsBf.All(math.MaxUint64)
+				if err != nil {
+					return nil, err
+				}
+
+				partitions1 := bitfield.New()
+				for i := 0; i < len(partitionsSlice)/2; i++ {
+					partitions1.Set(uint64(i))
+				}
+
+				msgs1, err := makeMsgForPartitions(partitions1)
+				if err != nil {
+					return nil, err
+				}
+
+				// time for the second half
+				partitions2 := bitfield.New()
+				for i := len(partitionsSlice) / 2; i < len(partitionsSlice); i++ {
+					partitions2.Set(uint64(i))
+				}
+
+				msgs2, err := makeMsgForPartitions(partitions2)
+				if err != nil {
+					return nil, err
+				}
+
+				return append(msgs1, msgs2...), nil
+			} else if err != nil {
+				return nil, err
+			}
+
+			return []*types.Message{estimatedMsg}, nil
+		}
+
 		partitions := bitfield.New()
 		for _, partition := range parts {
 			partitions.Set(uint64(partition))
 		}
 
-		params := miner.CompactPartitionsParams{
-			Deadline:   deadline,
-			Partitions: partitions,
-		}
-
-		sp, err := actors.SerializeParams(&params)
+		msgs, err := makeMsgForPartitions(partitions)
 		if err != nil {
-			return xerrors.Errorf("serializing params: %w", err)
+			return xerrors.Errorf("failed to make messages: %w", err)
 		}
 
-		smsg, err := api.MpoolPushMessage(ctx, &types.Message{
-			From:   minfo.Worker,
-			To:     maddr,
-			Method: builtin.MethodsMiner.CompactPartitions,
-			Value:  big.Zero(),
-			Params: sp,
-		}, nil)
-		if err != nil {
-			return xerrors.Errorf("mpool push: %w", err)
+		// Actually send the messages if really-do-it provided, simulate otherwise
+		if cctx.Bool("really-do-it") {
+			smsgs, err := fullNodeAPI.MpoolBatchPushMessage(ctx, msgs, nil)
+			if err != nil {
+				return xerrors.Errorf("mpool push: %w", err)
+			}
+
+			if len(smsgs) == 1 {
+				fmt.Printf("Requested compact partitions in message %s\n", smsgs[0].Cid())
+			} else {
+				fmt.Printf("Requested compact partitions in %d messages\n\n", len(smsgs))
+				for _, v := range smsgs {
+					fmt.Println(v.Cid())
+				}
+			}
+
+			for _, v := range smsgs {
+				wait, err := fullNodeAPI.StateWaitMsg(ctx, v.Cid(), 2)
+				if err != nil {
+					return err
+				}
+
+				// check it executed successfully
+				if wait.Receipt.ExitCode.IsError() {
+					fmt.Println(cctx.App.Writer, "compact partitions msg %s failed!", v.Cid())
+					return err
+				}
+			}
+
+			return nil
 		}
 
-		fmt.Printf("Requested compact partitions in message %s\n", smsg.Cid())
+		for i, v := range msgs {
+			fmt.Printf("total of %d CompactPartitions msgs would be sent\n", len(msgs))
 
-		wait, err := api.StateWaitMsg(ctx, smsg.Cid(), 0)
-		if err != nil {
-			return err
-		}
+			estMsg, err := fullNodeAPI.GasEstimateMessageGas(ctx, v, nil, types.EmptyTSK)
+			if err != nil {
+				return err
+			}
 
-		// check it executed successfully
-		if wait.Receipt.ExitCode.IsError() {
-			fmt.Println(cctx.App.Writer, "compact partitions failed!")
-			return err
+			fmt.Printf("msg %d would cost up to %s\n", i+1, types.FIL(estMsg.RequiredFunds()))
 		}
 
 		return nil
+
 	},
 }
 

cmd/lotus-miner/storage.go

@@ -225,6 +225,7 @@ var storageRedeclareCmd = &cli.Command{
 		&cli.BoolFlag{
 			Name:  "drop-missing",
 			Usage: "Drop index entries with missing files",
+			Value: true,
 		},
 	},
 	Action: func(cctx *cli.Context) error {
@@ -235,14 +236,19 @@ var storageRedeclareCmd = &cli.Command{
 		defer closer()
 		ctx := lcli.ReqContext(cctx)
 
-		if cctx.NArg() != 1 {
-			return lcli.IncorrectNumArgs(cctx)
+		// check if no argument and no --id or --all flag is provided
+		if cctx.NArg() == 0 && !cctx.IsSet("id") && !cctx.Bool("all") {
+			return xerrors.Errorf("You must specify a storage path, or --id, or --all")
 		}
 
 		if cctx.IsSet("id") && cctx.Bool("all") {
 			return xerrors.Errorf("--id and --all can't be passed at the same time")
 		}
 
+		if cctx.Bool("all") && cctx.NArg() > 0 {
+			return xerrors.Errorf("No additional arguments are expected when --all is set")
+		}
+
 		if cctx.IsSet("id") {
 			id := storiface.ID(cctx.String("id"))
 			return minerApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
@@ -252,7 +258,28 @@ var storageRedeclareCmd = &cli.Command{
 			return minerApi.StorageRedeclareLocal(ctx, nil, cctx.Bool("drop-missing"))
 		}
 
-		return xerrors.Errorf("either --all or --id must be specified")
+		// As no --id or --all flag is set, we can assume the argument is a path.
+		path := cctx.Args().First()
+		metaFilePath := filepath.Join(path, "sectorstore.json")
+
+		var meta storiface.LocalStorageMeta
+		metaFile, err := os.Open(metaFilePath)
+		if err != nil {
+			return xerrors.Errorf("Failed to open file: %w", err)
+		}
+		defer func() {
+			if closeErr := metaFile.Close(); closeErr != nil {
+				log.Error("Failed to close the file: %v", closeErr)
+			}
+		}()
+
+		err = json.NewDecoder(metaFile).Decode(&meta)
+		if err != nil {
+			return xerrors.Errorf("Failed to decode file: %w", err)
+		}
+
+		id := meta.ID
+		return minerApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
 	},
 }
 

cmd/lotus-shed/actor.go

@@ -968,7 +968,7 @@ var actorProposeChangeBeneficiary = &cli.Command{
 var actorConfirmChangeBeneficiary = &cli.Command{
 	Name:      "confirm-change-beneficiary",
 	Usage:     "Confirm a beneficiary address change",
-	ArgsUsage: "[minerAddress]",
+	ArgsUsage: "[minerID]",
 	Flags: []cli.Flag{
 		&cli.BoolFlag{
 			Name:  "really-do-it",

cmd/lotus-shed/eth.go

@@ -9,6 +9,7 @@ import (
 	"github.com/filecoin-project/go-state-types/abi"
 
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
 	lcli "github.com/filecoin-project/lotus/cli"
 )
 
@@ -23,6 +24,7 @@ var ethCmd = &cli.Command{
 	},
 	Subcommands: []*cli.Command{
 		checkTipsetsCmd,
+		computeEthHashCmd,
 	},
 }
 
@@ -70,3 +72,36 @@ var checkTipsetsCmd = &cli.Command{
 		return nil
 	},
 }
+
+var computeEthHashCmd = &cli.Command{
+	Name:  "compute-eth-hash",
+	Usage: "Compute the eth hash for a given message CID",
+	Action: func(cctx *cli.Context) error {
+		if cctx.NArg() != 1 {
+			return lcli.IncorrectNumArgs(cctx)
+		}
+
+		msg, err := messageFromString(cctx, cctx.Args().First())
+		if err != nil {
+			return err
+		}
+
+		switch msg := msg.(type) {
+		case *types.SignedMessage:
+			tx, err := ethtypes.EthTxFromSignedEthMessage(msg)
+			if err != nil {
+				return fmt.Errorf("failed to convert from signed message: %w", err)
+			}
+
+			tx.Hash, err = tx.TxHash()
+			if err != nil {
+				return fmt.Errorf("failed to call TxHash: %w", err)
+			}
+			fmt.Println(tx.Hash)
+		default:
+			return fmt.Errorf("not a signed message")
+		}
+
+		return nil
+	},
+}

cmd/lotus-shed/indexes.go

@@ -0,0 +1,337 @@
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"path"
+	"path/filepath"
+	"strings"
+
+	"github.com/mitchellh/go-homedir"
+	"github.com/urfave/cli/v2"
+	"golang.org/x/xerrors"
+
+	"github.com/filecoin-project/go-state-types/abi"
+	"github.com/filecoin-project/go-state-types/crypto"
+
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
+	lcli "github.com/filecoin-project/lotus/cli"
+)
+
+func withCategory(cat string, cmd *cli.Command) *cli.Command {
+	cmd.Category = strings.ToUpper(cat)
+	return cmd
+}
+
+var indexesCmd = &cli.Command{
+	Name:            "indexes",
+	Usage:           "Commands related to managing sqlite indexes",
+	HideHelpCommand: true,
+	Subcommands: []*cli.Command{
+		withCategory("msgindex", backfillMsgIndexCmd),
+		withCategory("msgindex", pruneMsgIndexCmd),
+		withCategory("txhash", backfillTxHashCmd),
+	},
+}
+
+var backfillMsgIndexCmd = &cli.Command{
+	Name:  "backfill-msgindex",
+	Usage: "Backfill the msgindex.db for a number of epochs starting from a specified height",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "from",
+			Value: 0,
+			Usage: "height to start the backfill; uses the current head if omitted",
+		},
+		&cli.IntFlag{
+			Name:  "epochs",
+			Value: 1800,
+			Usage: "number of epochs to backfill; defaults to 1800 (2 finalities)",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+
+		defer closer()
+		ctx := lcli.ReqContext(cctx)
+
+		curTs, err := api.ChainHead(ctx)
+		if err != nil {
+			return err
+		}
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight == 0 {
+			startHeight = int64(curTs.Height()) - 1
+		}
+		epochs := cctx.Int("epochs")
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		insertStmt, err := db.Prepare("INSERT OR IGNORE INTO messages (cid, tipset_cid, epoch) VALUES (?, ?, ?)")
+		if err != nil {
+			return err
+		}
+
+		var nrRowsAffected int64
+		for i := 0; i < epochs; i++ {
+			epoch := abi.ChainEpoch(startHeight - int64(i))
+
+			if i%100 == 0 {
+				log.Infof("%d/%d processing epoch:%d, nrRowsAffected:%d", i, epochs, epoch, nrRowsAffected)
+			}
+
+			ts, err := api.ChainGetTipSetByHeight(ctx, epoch, curTs.Key())
+			if err != nil {
+				return fmt.Errorf("failed to get tipset at epoch %d: %w", epoch, err)
+			}
+
+			tsCid, err := ts.Key().Cid()
+			if err != nil {
+				return fmt.Errorf("failed to get tipset cid at epoch %d: %w", epoch, err)
+			}
+
+			msgs, err := api.ChainGetMessagesInTipset(ctx, ts.Key())
+			if err != nil {
+				return fmt.Errorf("failed to get messages in tipset at epoch %d: %w", epoch, err)
+			}
+
+			for _, msg := range msgs {
+				key := msg.Cid.String()
+				tskey := tsCid.String()
+				res, err := insertStmt.Exec(key, tskey, int64(epoch))
+				if err != nil {
+					return fmt.Errorf("failed to insert message cid %s in tipset %s at epoch %d: %w", key, tskey, epoch, err)
+				}
+				rowsAffected, err := res.RowsAffected()
+				if err != nil {
+					return fmt.Errorf("failed to get rows affected for message cid %s in tipset %s at epoch %d: %w", key, tskey, epoch, err)
+				}
+				nrRowsAffected += rowsAffected
+			}
+		}
+
+		log.Infof("Done backfilling, nrRowsAffected:%d", nrRowsAffected)
+
+		return nil
+	},
+}
+
+var pruneMsgIndexCmd = &cli.Command{
+	Name:  "prune-msgindex",
+	Usage: "Prune the msgindex.db for messages included before a given epoch",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "from",
+			Usage: "height to start the prune; if negative it indicates epochs from current head",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+
+		defer closer()
+		ctx := lcli.ReqContext(cctx)
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight < 0 {
+			curTs, err := api.ChainHead(ctx)
+			if err != nil {
+				return err
+			}
+
+			startHeight += int64(curTs.Height())
+
+			if startHeight < 0 {
+				return xerrors.Errorf("bogus start height %d", startHeight)
+			}
+		}
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		tx, err := db.Begin()
+		if err != nil {
+			return err
+		}
+
+		if _, err := tx.Exec("DELETE FROM messages WHERE epoch < ?", startHeight); err != nil {
+			if err := tx.Rollback(); err != nil {
+				fmt.Printf("ERROR: rollback: %s", err)
+			}
+			return err
+		}
+
+		if err := tx.Commit(); err != nil {
+			return err
+		}
+
+		return nil
+	},
+}
+
+var backfillTxHashCmd = &cli.Command{
+	Name:  "backfill-txhash",
+	Usage: "Backfills the txhash.db for a number of epochs starting from a specified height",
+	Flags: []cli.Flag{
+		&cli.UintFlag{
+			Name:  "from",
+			Value: 0,
+			Usage: "the tipset height to start backfilling from (0 is head of chain)",
+		},
+		&cli.IntFlag{
+			Name:  "epochs",
+			Value: 2000,
+			Usage: "the number of epochs to backfill",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		ctx := lcli.ReqContext(cctx)
+
+		curTs, err := api.ChainHead(ctx)
+		if err != nil {
+			return err
+		}
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight == 0 {
+			startHeight = int64(curTs.Height()) - 1
+		}
+
+		epochs := cctx.Int("epochs")
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := filepath.Join(basePath, "sqlite", "txhash.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		insertStmt, err := db.Prepare("INSERT OR IGNORE INTO eth_tx_hashes(hash, cid) VALUES(?, ?)")
+		if err != nil {
+			return err
+		}
+
+		var totalRowsAffected int64 = 0
+		for i := 0; i < epochs; i++ {
+			epoch := abi.ChainEpoch(startHeight - int64(i))
+
+			select {
+			case <-cctx.Done():
+				fmt.Println("request cancelled")
+				return nil
+			default:
+			}
+
+			curTsk := curTs.Parents()
+			execTs, err := api.ChainGetTipSet(ctx, curTsk)
+			if err != nil {
+				return fmt.Errorf("failed to call ChainGetTipSet for %s: %w", curTsk, err)
+			}
+
+			if i%100 == 0 {
+				log.Infof("%d/%d processing epoch:%d", i, epochs, epoch)
+			}
+
+			for _, blockheader := range execTs.Blocks() {
+				blkMsgs, err := api.ChainGetBlockMessages(ctx, blockheader.Cid())
+				if err != nil {
+					log.Infof("Could not get block messages at epoch: %d, stopping walking up the chain", epoch)
+					epochs = i
+					break
+				}
+
+				for _, smsg := range blkMsgs.SecpkMessages {
+					if smsg.Signature.Type != crypto.SigTypeDelegated {
+						continue
+					}
+
+					tx, err := ethtypes.EthTxFromSignedEthMessage(smsg)
+					if err != nil {
+						return fmt.Errorf("failed to convert from signed message: %w at epoch: %d", err, epoch)
+					}
+
+					tx.Hash, err = tx.TxHash()
+					if err != nil {
+						return fmt.Errorf("failed to calculate hash for ethTx: %w at epoch: %d", err, epoch)
+					}
+
+					res, err := insertStmt.Exec(tx.Hash.String(), smsg.Cid().String())
+					if err != nil {
+						return fmt.Errorf("error inserting tx mapping to db: %s at epoch: %d", err, epoch)
+					}
+
+					rowsAffected, err := res.RowsAffected()
+					if err != nil {
+						return fmt.Errorf("error getting rows affected: %s at epoch: %d", err, epoch)
+					}
+
+					if rowsAffected > 0 {
+						log.Debugf("Inserted txhash %s, cid: %s at epoch: %d", tx.Hash.String(), smsg.Cid().String(), epoch)
+					}
+
+					totalRowsAffected += rowsAffected
+				}
+			}
+
+			curTs = execTs
+		}
+
+		log.Infof("Done, inserted %d missing txhashes", totalRowsAffected)
+
+		return nil
+	},
+}

cmd/lotus-shed/main.go

@@ -1,8 +1,10 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
+	"os/signal"
 
 	logging "github.com/ipfs/go-log/v2"
 	"github.com/urfave/cli/v2"
@@ -84,7 +86,7 @@ func main() {
 		invariantsCmd,
 		gasTraceCmd,
 		replayOfflineCmd,
-		msgindexCmd,
+		indexesCmd,
 		FevmAnalyticsCmd,
 		mismatchesCmd,
 	}
@@ -118,7 +120,20 @@ func main() {
 		},
 	}
 
-	if err := app.Run(os.Args); err != nil {
+	// terminate early on ctrl+c
+	c := make(chan os.Signal, 1)
+	signal.Notify(c, os.Interrupt)
+	ctx, cancel := context.WithCancel(context.Background())
+	go func() {
+		<-c
+		cancel()
+		fmt.Println("Received interrupt, shutting down... Press CTRL+C again to force shutdown")
+		<-c
+		fmt.Println("Forcing stop")
+		os.Exit(1)
+	}()
+
+	if err := app.RunContext(ctx, os.Args); err != nil {
 		log.Errorf("%+v", err)
 		os.Exit(1)
 		return

cmd/lotus-shed/msgindex.go

@@ -1,221 +0,0 @@
-package main
-
-import (
-	"database/sql"
-	"fmt"
-	"path"
-
-	"github.com/ipfs/go-cid"
-	_ "github.com/mattn/go-sqlite3"
-	"github.com/mitchellh/go-homedir"
-	"github.com/urfave/cli/v2"
-	"golang.org/x/xerrors"
-
-	"github.com/filecoin-project/go-state-types/abi"
-
-	lcli "github.com/filecoin-project/lotus/cli"
-)
-
-var msgindexCmd = &cli.Command{
-	Name:  "msgindex",
-	Usage: "Tools for managing the message index",
-	Subcommands: []*cli.Command{
-		msgindexBackfillCmd,
-		msgindexPruneCmd,
-	},
-}
-
-var msgindexBackfillCmd = &cli.Command{
-	Name:  "backfill",
-	Usage: "Backfill the message index for a number of epochs starting from a specified height",
-	Flags: []cli.Flag{
-		&cli.IntFlag{
-			Name:  "from",
-			Value: 0,
-			Usage: "height to start the backfill; uses the current head if omitted",
-		},
-		&cli.IntFlag{
-			Name:  "epochs",
-			Value: 1800,
-			Usage: "number of epochs to backfill; defaults to 1800 (2 finalities)",
-		},
-		&cli.StringFlag{
-			Name:  "repo",
-			Value: "~/.lotus",
-			Usage: "path to the repo",
-		},
-	},
-	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetFullNodeAPI(cctx)
-		if err != nil {
-			return err
-		}
-
-		defer closer()
-		ctx := lcli.ReqContext(cctx)
-
-		curTs, err := api.ChainHead(ctx)
-		if err != nil {
-			return err
-		}
-
-		startHeight := int64(cctx.Int("from"))
-		if startHeight == 0 {
-			startHeight = int64(curTs.Height()) - 1
-		}
-		epochs := cctx.Int("epochs")
-
-		basePath, err := homedir.Expand(cctx.String("repo"))
-		if err != nil {
-			return err
-		}
-
-		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
-		db, err := sql.Open("sqlite3", dbPath)
-		if err != nil {
-			return err
-		}
-
-		defer func() {
-			err := db.Close()
-			if err != nil {
-				fmt.Printf("ERROR: closing db: %s", err)
-			}
-		}()
-
-		tx, err := db.Begin()
-		if err != nil {
-			return err
-		}
-
-		insertStmt, err := tx.Prepare("INSERT INTO messages VALUES (?, ?, ?)")
-		if err != nil {
-			return err
-		}
-
-		insertMsg := func(cid, tsCid cid.Cid, epoch abi.ChainEpoch) error {
-			key := cid.String()
-			tskey := tsCid.String()
-			if _, err := insertStmt.Exec(key, tskey, int64(epoch)); err != nil {
-				return err
-			}
-
-			return nil
-		}
-		rollback := func() {
-			if err := tx.Rollback(); err != nil {
-				fmt.Printf("ERROR: rollback: %s", err)
-			}
-		}
-
-		for i := 0; i < epochs; i++ {
-			epoch := abi.ChainEpoch(startHeight - int64(i))
-
-			ts, err := api.ChainGetTipSetByHeight(ctx, epoch, curTs.Key())
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			tsCid, err := ts.Key().Cid()
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			msgs, err := api.ChainGetMessagesInTipset(ctx, ts.Key())
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			for _, msg := range msgs {
-				if err := insertMsg(msg.Cid, tsCid, epoch); err != nil {
-					rollback()
-					return err
-				}
-			}
-		}
-
-		if err := tx.Commit(); err != nil {
-			return err
-		}
-
-		return nil
-	},
-}
-
-var msgindexPruneCmd = &cli.Command{
-	Name:  "prune",
-	Usage: "Prune the message index for messages included before a given epoch",
-	Flags: []cli.Flag{
-		&cli.IntFlag{
-			Name:  "from",
-			Usage: "height to start the prune; if negative it indicates epochs from current head",
-		},
-		&cli.StringFlag{
-			Name:  "repo",
-			Value: "~/.lotus",
-			Usage: "path to the repo",
-		},
-	},
-	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetFullNodeAPI(cctx)
-		if err != nil {
-			return err
-		}
-
-		defer closer()
-		ctx := lcli.ReqContext(cctx)
-
-		startHeight := int64(cctx.Int("from"))
-		if startHeight < 0 {
-			curTs, err := api.ChainHead(ctx)
-			if err != nil {
-				return err
-			}
-
-			startHeight += int64(curTs.Height())
-
-			if startHeight < 0 {
-				return xerrors.Errorf("bogus start height %d", startHeight)
-			}
-		}
-
-		basePath, err := homedir.Expand(cctx.String("repo"))
-		if err != nil {
-			return err
-		}
-
-		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
-		db, err := sql.Open("sqlite3", dbPath)
-		if err != nil {
-			return err
-		}
-
-		defer func() {
-			err := db.Close()
-			if err != nil {
-				fmt.Printf("ERROR: closing db: %s", err)
-			}
-		}()
-
-		tx, err := db.Begin()
-		if err != nil {
-			return err
-		}
-
-		if _, err := tx.Exec("DELETE FROM messages WHERE epoch < ?", startHeight); err != nil {
-			if err := tx.Rollback(); err != nil {
-				fmt.Printf("ERROR: rollback: %s", err)
-			}
-			return err
-		}
-
-		if err := tx.Commit(); err != nil {
-			return err
-		}
-
-		return nil
-	},
-}

cmd/lotus-shed/sectors.go

@@ -653,7 +653,7 @@ fr32 padding is removed from the output.`,
 			return xerrors.Errorf("getting reader: %w", err)
 		}
 
-		rd, err := readStarter(0)
+		rd, err := readStarter(0, storiface.PaddedByteIndex(length))
 		if err != nil {
 			return xerrors.Errorf("starting reader: %w", err)
 		}

cmd/lotus-shed/verifreg.go

@@ -474,7 +474,7 @@ var verifRegRemoveVerifiedClientDataCapCmd = &cli.Command{
 
 		st, err := multisig.Load(store, vrkState)
 		if err != nil {
-			return err
+			return fmt.Errorf("load vrk failed: %w ", err)
 		}
 
 		signers, err := st.Signers()
@@ -508,14 +508,13 @@ var verifRegRemoveVerifiedClientDataCapCmd = &cli.Command{
 			return err
 		}
 
-		sm, _, err := srv.PublishMessage(ctx, proto, false)
+		sm, err := lcli.InteractiveSend(ctx, cctx, srv, proto)
 		if err != nil {
 			return err
 		}
 
 		msgCid := sm.Cid()
-
-		fmt.Printf("message sent, now waiting on cid: %s\n", msgCid)
+		fmt.Println("sending msg: ", msgCid)
 
 		mwait, err := api.StateWaitMsg(ctx, msgCid, uint64(cctx.Int("confidence")), build.Finality, true)
 		if err != nil {

cmd/lotus-worker/main.go

@@ -275,6 +275,12 @@ var runCmd = &cli.Command{
 			Name:  "http-server-timeout",
 			Value: "30s",
 		},
+		&cli.BoolFlag{
+			Name:        "data-cid",
+			Usage:       "Run the data-cid task. true|false",
+			Value:       true,
+			DefaultText: "inherits --addpiece",
+		},
 	},
 	Before: func(cctx *cli.Context) error {
 		if cctx.IsSet("address") {
@@ -386,8 +392,19 @@ var runCmd = &cli.Command{
 			}
 		}
 
+		ttDataCidDefault := false
 		if (workerType == sealtasks.WorkerSealing || cctx.IsSet("addpiece")) && cctx.Bool("addpiece") {
-			taskTypes = append(taskTypes, sealtasks.TTAddPiece, sealtasks.TTDataCid)
+			taskTypes = append(taskTypes, sealtasks.TTAddPiece)
+			ttDataCidDefault = true
+		}
+		if workerType == sealtasks.WorkerSealing {
+			if cctx.IsSet("data-cid") {
+				if cctx.Bool("data-cid") {
+					taskTypes = append(taskTypes, sealtasks.TTDataCid)
+				}
+			} else if ttDataCidDefault {
+				taskTypes = append(taskTypes, sealtasks.TTDataCid)
+			}
 		}
 		if (workerType == sealtasks.WorkerSealing || cctx.IsSet("sector-download")) && cctx.Bool("sector-download") {
 			taskTypes = append(taskTypes, sealtasks.TTDownloadSector)

cmd/lotus-worker/storage.go

@@ -178,6 +178,7 @@ var storageRedeclareCmd = &cli.Command{
 		&cli.BoolFlag{
 			Name:  "drop-missing",
 			Usage: "Drop index entries with missing files",
+			Value: true,
 		},
 	},
 	Action: func(cctx *cli.Context) error {
@@ -188,10 +189,19 @@ var storageRedeclareCmd = &cli.Command{
 		defer closer()
 		ctx := lcli.ReqContext(cctx)
 
+		// check if no argument and no --id or --all flag is provided
+		if cctx.NArg() == 0 && !cctx.IsSet("id") && !cctx.Bool("all") {
+			return xerrors.Errorf("You must specify a storage path, or --id, or --all")
+		}
+
 		if cctx.IsSet("id") && cctx.Bool("all") {
 			return xerrors.Errorf("--id and --all can't be passed at the same time")
 		}
 
+		if cctx.Bool("all") && cctx.NArg() > 0 {
+			return xerrors.Errorf("No additional arguments are expected when --all is set")
+		}
+
 		if cctx.IsSet("id") {
 			id := storiface.ID(cctx.String("id"))
 			return nodeApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
@@ -201,6 +211,27 @@ var storageRedeclareCmd = &cli.Command{
 			return nodeApi.StorageRedeclareLocal(ctx, nil, cctx.Bool("drop-missing"))
 		}
 
-		return xerrors.Errorf("either --all or --id must be specified")
+		// As no --id or --all flag is set, we can assume the argument is a path.
+		path := cctx.Args().First()
+		metaFilePath := filepath.Join(path, "sectorstore.json")
+
+		var meta storiface.LocalStorageMeta
+		metaFile, err := os.Open(metaFilePath)
+		if err != nil {
+			return xerrors.Errorf("Failed to open file: %w", err)
+		}
+		defer func() {
+			if closeErr := metaFile.Close(); closeErr != nil {
+				log.Error("Failed to close the file: %v", closeErr)
+			}
+		}()
+
+		err = json.NewDecoder(metaFile).Decode(&meta)
+		if err != nil {
+			return xerrors.Errorf("Failed to decode file: %w", err)
+		}
+
+		id := meta.ID
+		return nodeApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
 	},
 }

cmd/lotus/daemon.go

@@ -16,9 +16,12 @@ import (
 	"strings"
 
 	"github.com/DataDog/zstd"
+	"github.com/ipfs/go-cid"
+	levelds "github.com/ipfs/go-ds-leveldb"
 	metricsprom "github.com/ipfs/go-metrics-prometheus"
 	"github.com/mitchellh/go-homedir"
 	"github.com/multiformats/go-multiaddr"
+	ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
 	"github.com/urfave/cli/v2"
 	"go.opencensus.io/plugin/runmetrics"
 	"go.opencensus.io/stats"
@@ -27,13 +30,19 @@ import (
 	"golang.org/x/xerrors"
 	"gopkg.in/cheggaaa/pb.v1"
 
+	"github.com/filecoin-project/go-address"
+	cborutil "github.com/filecoin-project/go-cbor-util"
 	"github.com/filecoin-project/go-jsonrpc"
 	"github.com/filecoin-project/go-paramfetch"
+	"github.com/filecoin-project/go-state-types/builtin"
+	"github.com/filecoin-project/specs-actors/actors/builtin/miner"
 
 	lapi "github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/chain/actors"
 	"github.com/filecoin-project/lotus/chain/consensus"
 	"github.com/filecoin-project/lotus/chain/consensus/filcns"
+	"github.com/filecoin-project/lotus/chain/gen/slashfilter"
 	"github.com/filecoin-project/lotus/chain/index"
 	"github.com/filecoin-project/lotus/chain/stmgr"
 	"github.com/filecoin-project/lotus/chain/store"
@@ -160,6 +169,19 @@ var DaemonCmd = &cli.Command{
 			Name:  "restore-config",
 			Usage: "config file to use when restoring from backup",
 		},
+		&cli.BoolFlag{
+			Name:  "slash-consensus",
+			Usage: "Report consensus fault",
+			Value: false,
+		},
+		&cli.StringFlag{
+			Name:  "slasher-sender",
+			Usage: "optionally specify the account to report consensus from",
+		},
+		&cli.StringFlag{
+			Name:  "slashdb-dir",
+			Value: "slash watch db dir path",
+		},
 	},
 	Action: func(cctx *cli.Context) error {
 		isLite := cctx.Bool("lite")
@@ -380,7 +402,14 @@ var DaemonCmd = &cli.Command{
 		if err != nil {
 			return fmt.Errorf("failed to start json-rpc endpoint: %s", err)
 		}
-
+		if cctx.IsSet("slash-consensus") && cctx.IsSet("slashdb-dir") {
+			go func() {
+				err := slashConsensus(api, cctx.String("slashdb-dir"), cctx.String("slasher-sender"))
+				if err != nil {
+					panic("slashConsensus error")
+				}
+			}()
+		}
 		// Monitor for shutdown.
 		finishCh := node.MonitorShutdown(shutdownChan,
 			node.ShutdownHandler{Component: "rpc server", StopFunc: rpcStopper},
@@ -574,3 +603,122 @@ func ImportChain(ctx context.Context, r repo.Repo, fname string, snapshot bool)
 
 	return nil
 }
+
+func slashConsensus(a lapi.FullNode, p string, from string) error {
+	ctx := context.Background()
+	var fromAddr address.Address
+
+	ds, err := levelds.NewDatastore(p, &levelds.Options{
+		Compression: ldbopts.NoCompression,
+		NoSync:      false,
+		Strict:      ldbopts.StrictAll,
+		ReadOnly:    false,
+	})
+	if err != nil {
+		return xerrors.Errorf("open leveldb: %w", err)
+	}
+	sf := slashfilter.New(ds)
+	if from == "" {
+		defaddr, err := a.WalletDefaultAddress(ctx)
+		if err != nil {
+			return err
+		}
+		fromAddr = defaddr
+	} else {
+		addr, err := address.NewFromString(from)
+		if err != nil {
+			return err
+		}
+
+		fromAddr = addr
+	}
+
+	blocks, err := a.SyncIncomingBlocks(ctx)
+	if err != nil {
+		return xerrors.Errorf("sync incoming blocks failed: %w", err)
+	}
+	for block := range blocks {
+		log.Infof("deal with block: %d, %v, %s", block.Height, block.Miner, block.Cid())
+		if otherBlock, extraBlock, err := slashFilterMinedBlock(ctx, sf, a, block); err != nil {
+			if otherBlock == nil {
+				continue
+			}
+			log.Errorf("<!!> SLASH FILTER ERROR: %s", err)
+			bh1, err := cborutil.Dump(otherBlock)
+			if err != nil {
+				log.Errorf("could not dump otherblock:%s, err:%s", otherBlock.Cid(), err)
+				continue
+			}
+
+			bh2, err := cborutil.Dump(block)
+			if err != nil {
+				log.Errorf("could not dump block:%s, err:%s", block.Cid(), err)
+				continue
+			}
+
+			params := miner.ReportConsensusFaultParams{
+				BlockHeader1: bh1,
+				BlockHeader2: bh2,
+			}
+			if extraBlock != nil {
+				be, err := cborutil.Dump(extraBlock)
+				if err != nil {
+					log.Errorf("could not dump block:%s, err:%s", block.Cid(), err)
+					continue
+				}
+				params.BlockHeaderExtra = be
+			}
+
+			enc, err := actors.SerializeParams(&params)
+			if err != nil {
+				log.Errorf("could not serialize declare faults parameters: %s", err)
+				continue
+			}
+			message, err := a.MpoolPushMessage(ctx, &types.Message{
+				To:     block.Miner,
+				From:   fromAddr,
+				Value:  types.NewInt(0),
+				Method: builtin.MethodsMiner.ReportConsensusFault,
+				Params: enc,
+			}, nil)
+			if err != nil {
+				log.Errorf("ReportConsensusFault to messagepool error:%w", err)
+				continue
+			}
+			log.Infof("ReportConsensusFault message CID:%s", message.Cid())
+
+		}
+	}
+	return err
+}
+
+func slashFilterMinedBlock(ctx context.Context, sf *slashfilter.SlashFilter, a lapi.FullNode, blockB *types.BlockHeader) (*types.BlockHeader, *types.BlockHeader, error) {
+	blockC, err := a.ChainGetBlock(ctx, blockB.Parents[0])
+	if err != nil {
+		return nil, nil, xerrors.Errorf("chain get block error:%s", err)
+	}
+	otherCid, err := sf.MinedBlock(ctx, blockB, blockC.Height)
+	if err != nil {
+		return nil, nil, xerrors.Errorf("slash filter check block error:%s", err)
+	}
+	if otherCid != cid.Undef {
+		otherHeader, err := a.ChainGetBlock(ctx, otherCid)
+		return otherHeader, nil, xerrors.Errorf("chain get other block error:%s", err)
+	}
+	blockA, err := a.ChainGetBlock(ctx, otherCid)
+
+	// (c) parent-grinding fault
+	// Here extra is the "witness", a third block that shows the connection between A and B as
+	// A's sibling and B's parent.
+	// Specifically, since A is of lower height, it must be that B was mined omitting A from its tipset
+	//
+	//      B
+	//      |
+	//  [A, C]
+	if types.CidArrsEqual(blockA.Parents, blockC.Parents) && blockA.Height == blockC.Height &&
+		types.CidArrsContains(blockB.Parents, blockC.Cid()) && !types.CidArrsContains(blockB.Parents, blockA.Cid()) {
+		return blockA, blockC, xerrors.Errorf("chain get other block error:%s", err)
+	}
+
+	return nil, nil, nil
+}

documentation/en/api-v0-methods.md

@@ -4202,7 +4202,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -4233,7 +4233,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -4953,7 +4953,7 @@ Response:
     },
     "Nonce": 42,
     "Balance": "0",
-    "Address": "\u003cempty\u003e"
+    "Address": "f01234"
   }
 }
 ```
@@ -5242,7 +5242,7 @@ Response:
   },
   "Nonce": 42,
   "Balance": "0",
-  "Address": "\u003cempty\u003e"
+  "Address": "f01234"
 }
 ```
 
@@ -5430,7 +5430,6 @@ Response:
     "UpgradeRefuelHeight": 10101,
     "UpgradeTapeHeight": 10101,
     "UpgradeKumquatHeight": 10101,
-    "UpgradePriceListOopsHeight": 10101,
     "BreezeGasTampingDuration": 10101,
     "UpgradeCalicoHeight": 10101,
     "UpgradePersianHeight": 10101,
@@ -5986,6 +5985,7 @@ Response:
   "SectorSize": 34359738368,
   "WindowPoStPartitionSectors": 42,
   "ConsensusFaultElapsed": 10101,
+  "PendingOwnerAddress": "f01234",
   "Beneficiary": "f01234",
   "BeneficiaryTerm": {
     "Quota": "0",

documentation/en/api-v1-unstable-methods.md

@@ -103,6 +103,7 @@
   * [EthProtocolVersion](#EthProtocolVersion)
   * [EthSendRawTransaction](#EthSendRawTransaction)
   * [EthSubscribe](#EthSubscribe)
+  * [EthSyncing](#EthSyncing)
   * [EthUninstallFilter](#EthUninstallFilter)
   * [EthUnsubscribe](#EthUnsubscribe)
 * [Filecoin](#Filecoin)
@@ -2598,7 +2599,7 @@ Polling method for a filter, returns event logs which occurred since last poll.
 (requires write perm since timestamp of last filter execution will be written)
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -2619,7 +2620,7 @@ Returns event logs matching filter with given id.
 (requires write perm since timestamp of last filter execution will be written)
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -2990,7 +2991,7 @@ Response: `"0x0"`
 Installs a persistent filter to notify when a new block arrives.
 
 
-Perms: write
+Perms: read
 
 Inputs: `null`
 
@@ -3000,7 +3001,7 @@ Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 Installs a persistent filter based on given filter spec.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3021,7 +3022,7 @@ Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 Installs a persistent filter to notify when new messages arrive in the message pool.
 
 
-Perms: write
+Perms: read
 
 Inputs: `null`
 
@@ -3060,7 +3061,7 @@ params contains additional parameters used with the log event type
 The client will receive a stream of EthSubscriptionResponse values until EthUnsubscribe is called.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3071,11 +3072,20 @@ Inputs:
 
 Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 
+### EthSyncing
+
+
+Perms: read
+
+Inputs: `null`
+
+Response: `false`
+
 ### EthUninstallFilter
 Uninstalls a filter with given id.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3090,7 +3100,7 @@ Response: `true`
 Unsubscribe from a websocket subscription
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -5566,7 +5576,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -5597,7 +5607,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -6380,7 +6390,7 @@ Response:
     },
     "Nonce": 42,
     "Balance": "0",
-    "Address": "\u003cempty\u003e"
+    "Address": "f01234"
   }
 }
 ```
@@ -6720,7 +6730,7 @@ Response:
   },
   "Nonce": 42,
   "Balance": "0",
-  "Address": "\u003cempty\u003e"
+  "Address": "f01234"
 }
 ```
 
@@ -6932,7 +6942,6 @@ Response:
     "UpgradeRefuelHeight": 10101,
     "UpgradeTapeHeight": 10101,
     "UpgradeKumquatHeight": 10101,
-    "UpgradePriceListOopsHeight": 10101,
     "BreezeGasTampingDuration": 10101,
     "UpgradeCalicoHeight": 10101,
     "UpgradePersianHeight": 10101,
@@ -7496,6 +7505,7 @@ Response:
   "SectorSize": 34359738368,
   "WindowPoStPartitionSectors": 42,
   "ConsensusFaultElapsed": 10101,
+  "PendingOwnerAddress": "f01234",
   "Beneficiary": "f01234",
   "BeneficiaryTerm": {
     "Quota": "0",

documentation/en/cli-lotus-worker.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-worker [global options] command [command options] [arguments...]
 
 VERSION:
-   1.23.1-rc4
+   1.23.2-dev
 
 COMMANDS:
    run        Start lotus worker
@@ -19,12 +19,11 @@ COMMANDS:
    help, h    Shows a list of commands or help for one command
 
 GLOBAL OPTIONS:
-   --enable-gpu-proving                     enable use of GPU for mining operations (default: true) [$LOTUS_WORKER_ENABLE_GPU_PROVING]
-   --help, -h                               show help (default: false)
-   --miner-repo value, --storagerepo value  Specify miner repo path. flag storagerepo and env LOTUS_STORAGE_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
-   --version, -v                            print the version (default: false)
    --worker-repo value, --workerrepo value  Specify worker repo path. flag workerrepo and env WORKER_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusworker") [$LOTUS_WORKER_PATH, $WORKER_PATH]
-   
+   --miner-repo value, --storagerepo value  Specify miner repo path. flag storagerepo and env LOTUS_STORAGE_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
+   --enable-gpu-proving                     enable use of GPU for mining operations (default: true) [$LOTUS_WORKER_ENABLE_GPU_PROVING]
+   --help, -h                               show help
+   --version, -v                            print the version
 ```
 
 ## lotus-worker run
@@ -36,28 +35,29 @@ USAGE:
    lotus-worker run [command options] [arguments...]
 
 OPTIONS:
-   --addpiece                    enable addpiece (default: true) [$LOTUS_WORKER_ADDPIECE]
-   --commit                      enable commit (default: true) [$LOTUS_WORKER_COMMIT]
-   --http-server-timeout value   (default: "30s")
    --listen value                host address and port the worker api will listen on (default: "0.0.0.0:3456") [$LOTUS_WORKER_LISTEN]
-   --name value                  custom worker name (default: hostname) [$LOTUS_WORKER_NAME]
-   --no-default                  disable all default compute tasks, use the worker for storage/fetching only (default: false) [$LOTUS_WORKER_NO_DEFAULT]
    --no-local-storage            don't use storageminer repo for sector storage (default: false) [$LOTUS_WORKER_NO_LOCAL_STORAGE]
    --no-swap                     don't use swap (default: false) [$LOTUS_WORKER_NO_SWAP]
+   --name value                  custom worker name (default: hostname) [$LOTUS_WORKER_NAME]
+   --addpiece                    enable addpiece (default: true) [$LOTUS_WORKER_ADDPIECE]
+   --precommit1                  enable precommit1 (default: true) [$LOTUS_WORKER_PRECOMMIT1]
+   --unseal                      enable unsealing (default: true) [$LOTUS_WORKER_UNSEAL]
+   --precommit2                  enable precommit2 (default: true) [$LOTUS_WORKER_PRECOMMIT2]
+   --commit                      enable commit (default: true) [$LOTUS_WORKER_COMMIT]
+   --replica-update              enable replica update (default: true) [$LOTUS_WORKER_REPLICA_UPDATE]
+   --prove-replica-update2       enable prove replica update 2 (default: true) [$LOTUS_WORKER_PROVE_REPLICA_UPDATE2]
+   --regen-sector-key            enable regen sector key (default: true) [$LOTUS_WORKER_REGEN_SECTOR_KEY]
+   --sector-download             enable external sector data download (default: false) [$LOTUS_WORKER_SECTOR_DOWNLOAD]
+   --windowpost                  enable window post (default: false) [$LOTUS_WORKER_WINDOWPOST]
+   --winningpost                 enable winning post (default: false) [$LOTUS_WORKER_WINNINGPOST]
+   --no-default                  disable all default compute tasks, use the worker for storage/fetching only (default: false) [$LOTUS_WORKER_NO_DEFAULT]
    --parallel-fetch-limit value  maximum fetch operations to run in parallel (default: 5) [$LOTUS_WORKER_PARALLEL_FETCH_LIMIT]
    --post-parallel-reads value   maximum number of parallel challenge reads (0 = no limit) (default: 32) [$LOTUS_WORKER_POST_PARALLEL_READS]
    --post-read-timeout value     time limit for reading PoSt challenges (0 = no limit) (default: 0s) [$LOTUS_WORKER_POST_READ_TIMEOUT]
-   --precommit1                  enable precommit1 (default: true) [$LOTUS_WORKER_PRECOMMIT1]
-   --precommit2                  enable precommit2 (default: true) [$LOTUS_WORKER_PRECOMMIT2]
-   --prove-replica-update2       enable prove replica update 2 (default: true) [$LOTUS_WORKER_PROVE_REPLICA_UPDATE2]
-   --regen-sector-key            enable regen sector key (default: true) [$LOTUS_WORKER_REGEN_SECTOR_KEY]
-   --replica-update              enable replica update (default: true) [$LOTUS_WORKER_REPLICA_UPDATE]
-   --sector-download             enable external sector data download (default: false) [$LOTUS_WORKER_SECTOR_DOWNLOAD]
    --timeout value               used when 'listen' is unspecified. must be a valid duration recognized by golang's time.ParseDuration function (default: "30m") [$LOTUS_WORKER_TIMEOUT]
-   --unseal                      enable unsealing (default: true) [$LOTUS_WORKER_UNSEAL]
-   --windowpost                  enable window post (default: false) [$LOTUS_WORKER_WINDOWPOST]
-   --winningpost                 enable winning post (default: false) [$LOTUS_WORKER_WINNINGPOST]
-   
+   --http-server-timeout value   (default: "30s")
+   --data-cid                    Run the data-cid task. true|false (default: inherits --addpiece)
+   --help, -h                    show help
 ```
 
 ## lotus-worker stop
@@ -69,8 +69,7 @@ USAGE:
    lotus-worker stop [command options] [arguments...]
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ## lotus-worker info
@@ -82,8 +81,7 @@ USAGE:
    lotus-worker info [command options] [arguments...]
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ## lotus-worker storage
@@ -95,14 +93,13 @@ USAGE:
    lotus-worker storage command [command options] [arguments...]
 
 COMMANDS:
-     attach     attach local storage path
-     detach     detach local storage path
-     redeclare  redeclare sectors in a local storage path
-     help, h    Shows a list of commands or help for one command
+   attach     attach local storage path
+   detach     detach local storage path
+   redeclare  redeclare sectors in a local storage path
+   help, h    Shows a list of commands or help for one command
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ### lotus-worker storage attach
@@ -114,14 +111,14 @@ USAGE:
    lotus-worker storage attach [command options] [arguments...]
 
 OPTIONS:
-   --allow-to value [ --allow-to value ]  path groups allowed to pull data from this path (allow all if not specified)
-   --groups value [ --groups value ]      path group names
    --init                                 initialize the path first (default: false)
-   --max-storage value                    (for init) limit storage space for sectors (expensive for very large paths!)
+   --weight value                         (for init) path weight (default: 10)
    --seal                                 (for init) use path for sealing (default: false)
    --store                                (for init) use path for long-term storage (default: false)
-   --weight value                         (for init) path weight (default: 10)
-   
+   --max-storage value                    (for init) limit storage space for sectors (expensive for very large paths!)
+   --groups value [ --groups value ]      path group names
+   --allow-to value [ --allow-to value ]  path groups allowed to pull data from this path (allow all if not specified)
+   --help, -h                             show help
 ```
 
 ### lotus-worker storage detach
@@ -134,7 +131,7 @@ USAGE:
 
 OPTIONS:
    --really-do-it  (default: false)
-   
+   --help, -h      show help
 ```
 
 ### lotus-worker storage redeclare
@@ -146,10 +143,10 @@ USAGE:
    lotus-worker storage redeclare [command options] [arguments...]
 
 OPTIONS:
-   --all           redeclare all storage paths (default: false)
-   --drop-missing  Drop index entries with missing files (default: false)
    --id value      storage path ID
-   
+   --all           redeclare all storage paths (default: false)
+   --drop-missing  Drop index entries with missing files (default: true)
+   --help, -h      show help
 ```
 
 ## lotus-worker resources
@@ -161,9 +158,9 @@ USAGE:
    lotus-worker resources [command options] [arguments...]
 
 OPTIONS:
-   --all      print all resource envvars (default: false)
-   --default  print default resource envvars (default: false)
-   
+   --all       print all resource envvars (default: false)
+   --default   print default resource envvars (default: false)
+   --help, -h  show help
 ```
 
 ## lotus-worker tasks
@@ -175,13 +172,12 @@ USAGE:
    lotus-worker tasks command [command options] [arguments...]
 
 COMMANDS:
-     enable   Enable a task type
-     disable  Disable a task type
-     help, h  Shows a list of commands or help for one command
+   enable   Enable a task type
+   disable  Disable a task type
+   help, h  Shows a list of commands or help for one command
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ### lotus-worker tasks enable
@@ -193,8 +189,8 @@ USAGE:
    lotus-worker tasks enable [command options] --all | [UNS|C2|PC2|PC1|PR2|RU|AP|DC|GSK]
 
 OPTIONS:
-   --all  Enable all task types (default: false)
-   
+   --all       Enable all task types (default: false)
+   --help, -h  show help
 ```
 
 ### lotus-worker tasks disable
@@ -206,6 +202,6 @@ USAGE:
    lotus-worker tasks disable [command options] --all | [UNS|C2|PC2|PC1|PR2|RU|AP|DC|GSK]
 
 OPTIONS:
-   --all  Disable all task types (default: false)
-   
+   --all       Disable all task types (default: false)
+   --help, -h  show help
 ```

documentation/en/default-lotus-config.toml

@@ -38,7 +38,7 @@
   #
   # type: []string
   # env var: LOTUS_LIBP2P_LISTENADDRESSES
-  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0"]
+  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0", "/ip4/0.0.0.0/udp/0/quic-v1", "/ip6/::/udp/0/quic-v1", "/ip4/0.0.0.0/udp/0/quic-v1/webtransport", "/ip6/::/udp/0/quic-v1/webtransport"]
 
   # Addresses to explicitally announce to other peers. If not specified,
   # all interface addresses are announced
@@ -391,6 +391,7 @@
 
 
 [Index]
+  # EXPERIMENTAL FEATURE. USE WITH CAUTION
   # EnableMsgIndex enables indexing of messages on chain.
   #
   # type: bool

documentation/en/default-lotus-miner-config.toml

@@ -38,7 +38,7 @@
   #
   # type: []string
   # env var: LOTUS_LIBP2P_LISTENADDRESSES
-  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0"]
+  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0", "/ip4/0.0.0.0/udp/0/quic-v1", "/ip6/::/udp/0/quic-v1", "/ip4/0.0.0.0/udp/0/quic-v1/webtransport", "/ip6/::/udp/0/quic-v1/webtransport"]
 
   # Addresses to explicitally announce to other peers. If not specified,
   # all interface addresses are announced
@@ -515,15 +515,7 @@
   # env var: LOTUS_SEALING_MINUPGRADESECTOREXPIRATION
   #MinUpgradeSectorExpiration = 0
 
-  # When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-  # be selected based on lowest initial pledge.
-  # 
-  # Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-  # selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-  # where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-  # 
-  # Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-  # initial pledge - upgrade sectors will always be chosen based on longest expiration
+  # DEPRECATED: Target expiration is no longer used
   #
   # type: uint64
   # env var: LOTUS_SEALING_MINTARGETUPGRADESECTOREXPIRATION

documentation/misc/gas_balancing.md

@@ -2,8 +2,8 @@
 
 The gas balancing process targets to set gas costs of syscalls to be in line with
 10 gas per nanosecond on reference hardware.
-The process can be either performed for all syscalls based on existing messages and chain or targeted
-at single syscall.
+The process can be either performed for all syscalls based on existing messages and chains or targeted
+at a single syscall.
 
 #### Reference hardware
 
@@ -12,14 +12,14 @@ may be subject to change.
 
 ### Complete gas balancing
 
-Complete gas balancing is performed using `lotus-bench` the process is based on importing a chain export
+Complete gas balancing is performed using a `lotus-bench` the process is based on importing a chain export
 and collecting gas traces which are later aggregated.
 
 Before building `lotus-bench` make sure `EnableDetailedTracing` in `chain/vm/runtime.go` is set to `true`.
 
 The process can be started using `./lotus-bench import` with `--car` flag set to the location of
-CAR chain export. `--start-epoch` and `--end-epoch` can be used to to limit the range of epochs to run
-the benchmark. Note that state tree of `start-epoch` needs to be in the CAR file or has to be previously computed
+CAR chain export. `--start-epoch` and `--end-epoch` can be used to limit the range of epochs to run
+the benchmark. Note that the state tree of `start-epoch` needs to be in the CAR file or has to be previously computed
 to work.
 
 The output will be a `bench.json` file containing information about every syscall invoked
@@ -29,7 +29,7 @@ spare space.
 After the bench run is complete the `bench.json` file can be analyzed with `./lotus-bench import analyze bench.json`.
 
 It will compute means, standard deviations and co-variances (when applicable) of syscall runtimes. 
-The output is in nanoseconds, so the gas values for syscalls should be 10x that. In cases where co-variance of
+The output is in nanoseconds, so the gas values for syscalls should be 10x that. In cases where the co-variance of
 execution time to some parameter is evaluated, the strength of the correlation should be taken into account.
 
 #### Special cases
@@ -40,15 +40,15 @@ during block execution (when gas traces are formed) objects are only written to
 
 ### Targeted gas balancing
 
-In some cases complete gas balancing is infeasible, either new syscall gets introduced or
+In some cases complete gas balancing is infeasible, either a new syscall gets introduced or
 complete balancing is too time consuming.
 
-In these cases the recommended way to estimate gas for given syscall is to perform an `in-vivo` benchmark.
+In these cases, the recommended way to estimate gas for a given syscall is to perform an `in-vivo` benchmark.
 In the past `in-vitro` as in standalone benchmarks were found to be highly inaccurate when compared to results
 of real execution.
 
-A in-vivo benchmark can be performed by running an example of such syscall during block execution.
-The best place to hook-in such benchmark is message execution loop in
-`chain/stmgr/stmgr.go` in `ApplyBlocks()`. Depending of time required to complete the syscall it might be
+An in-vivo benchmark can be performed by running an example of such a syscall during block execution.
+The best place to hook-in such a benchmark is the message execution loop in
+`chain/stmgr/stmgr.go` in `ApplyBlocks()`. Depending on the time required to complete the syscall it might be
 advisable to run the execution only once every few messages.
 

gateway/node.go

@@ -43,6 +43,9 @@ const (
 // TargetAPI defines the API methods that the Node depends on
 // (to make it easy to mock for tests)
 type TargetAPI interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*api.MiningBaseInfo, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*api.InvocResult, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (api.MinerSectors, error)
@@ -115,6 +118,7 @@ type TargetAPI interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error)
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)
 	NetVersion(ctx context.Context) (string, error)
 	NetListening(ctx context.Context) (bool, error)
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)

gateway/proxy_eth.go

@@ -252,6 +252,14 @@ func (gw *Node) EthChainId(ctx context.Context) (ethtypes.EthUint64, error) {
 	return gw.target.EthChainId(ctx)
 }
 
+func (gw *Node) EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error) {
+	if err := gw.limit(ctx, basicRateLimitTokens); err != nil {
+		return ethtypes.EthSyncingResult{}, err
+	}
+
+	return gw.target.EthSyncing(ctx)
+}
+
 func (gw *Node) NetVersion(ctx context.Context) (string, error) {
 	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
 		return "", err

gateway/proxy_fil.go

@@ -23,6 +23,33 @@ import (
 	"github.com/filecoin-project/lotus/node/modules/dtypes"
 )
 
+func (gw *Node) MpoolPending(ctx context.Context, tsk types.TipSetKey) ([]*types.SignedMessage, error) {
+	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
+		return nil, err
+	}
+	if err := gw.checkTipsetKey(ctx, tsk); err != nil {
+		return nil, err
+	}
+	return gw.target.MpoolPending(ctx, tsk)
+}
+
+func (gw *Node) ChainGetBlock(ctx context.Context, c cid.Cid) (*types.BlockHeader, error) {
+	if err := gw.limit(ctx, chainRateLimitTokens); err != nil {
+		return nil, err
+	}
+	return gw.target.ChainGetBlock(ctx, c)
+}
+
+func (gw *Node) MinerGetBaseInfo(ctx context.Context, addr address.Address, h abi.ChainEpoch, tsk types.TipSetKey) (*api.MiningBaseInfo, error) {
+	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
+		return nil, err
+	}
+	if err := gw.checkTipsetKey(ctx, tsk); err != nil {
+		return nil, err
+	}
+	return gw.target.MinerGetBaseInfo(ctx, addr, h, tsk)
+}
+
 func (gw *Node) StateReplay(ctx context.Context, tsk types.TipSetKey, c cid.Cid) (*api.InvocResult, error) {
 	if err := gw.limit(ctx, chainRateLimitTokens); err != nil {
 		return nil, err

go.mod

@@ -38,11 +38,11 @@ require (
 	github.com/filecoin-project/go-cbor-util v0.0.1
 	github.com/filecoin-project/go-commp-utils v0.1.3
 	github.com/filecoin-project/go-crypto v0.0.1
-	github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4
+	github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7
 	github.com/filecoin-project/go-fil-commcid v0.1.0
 	github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
-	github.com/filecoin-project/go-fil-markets v1.27.0-rc1
-	github.com/filecoin-project/go-jsonrpc v0.2.3
+	github.com/filecoin-project/go-fil-markets v1.28.2
+	github.com/filecoin-project/go-jsonrpc v0.3.1
 	github.com/filecoin-project/go-padreader v0.0.1
 	github.com/filecoin-project/go-paramfetch v0.0.4
 	github.com/filecoin-project/go-state-types v0.11.1
@@ -87,7 +87,7 @@ require (
 	github.com/ipfs/go-ds-leveldb v0.5.0
 	github.com/ipfs/go-ds-measure v0.2.0
 	github.com/ipfs/go-fs-lock v0.0.7
-	github.com/ipfs/go-graphsync v0.14.3
+	github.com/ipfs/go-graphsync v0.14.6
 	github.com/ipfs/go-ipfs-blockstore v1.3.0
 	github.com/ipfs/go-ipfs-blocksutil v0.0.1
 	github.com/ipfs/go-ipfs-chunker v0.0.5
@@ -111,8 +111,8 @@ require (
 	github.com/ipld/go-codec-dagpb v1.6.0
 	github.com/ipld/go-ipld-prime v0.20.0
 	github.com/ipld/go-ipld-selector-text-lite v0.0.1
-	github.com/ipni/index-provider v0.11.0
-	github.com/ipni/storetheindex v0.5.10
+	github.com/ipni/go-libipni v0.0.8
+	github.com/ipni/index-provider v0.12.0
 	github.com/kelseyhightower/envconfig v1.4.0
 	github.com/koalacxr/quantile v0.0.1
 	github.com/libp2p/go-buffer-pool v0.1.0
@@ -144,7 +144,7 @@ require (
 	github.com/raulk/go-watchdog v1.3.0
 	github.com/stretchr/testify v1.8.2
 	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
-	github.com/urfave/cli/v2 v2.16.3
+	github.com/urfave/cli/v2 v2.25.5
 	github.com/whyrusleeping/bencher v0.0.0-20190829221104-bb6607aa8bba
 	github.com/whyrusleeping/cbor-gen v0.0.0-20230126041949-52956bd4c9aa
 	github.com/whyrusleeping/ledger-filecoin-go v0.9.1-0.20201010031517-c3dcc1bddce4
@@ -286,7 +286,7 @@ require (
 	github.com/mr-tron/base58 v1.2.0 // indirect
 	github.com/multiformats/go-base36 v0.2.0 // indirect
 	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
-	github.com/multiformats/go-multicodec v0.8.1 // indirect
+	github.com/multiformats/go-multicodec v0.9.0 // indirect
 	github.com/multiformats/go-multistream v0.4.1 // indirect
 	github.com/nikkolasg/hexjson v0.1.0 // indirect
 	github.com/nkovacs/streamquote v1.0.0 // indirect

go.sum

@@ -312,8 +312,8 @@ github.com/filecoin-project/go-commp-utils/nonffi v0.0.0-20220905160352-62059082
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
 github.com/filecoin-project/go-crypto v0.0.1 h1:AcvpSGGCgjaY8y1az6AMfKQWreF/pWO2JJGLl6gCq6o=
 github.com/filecoin-project/go-crypto v0.0.1/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
-github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4 h1:Y5RMvFT4OthsAhDx7xKfkJ5QdiWq9Ox7N46Mi0PF0PE=
-github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4/go.mod h1:1WDoUgWYB2KvogfPTdDmoyBTa9cb9+oGwqKCCipnJeY=
+github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7 h1:v+zJS5B6pA3ptWZS4t8tbt1Hz9qENnN4nVr1w99aSWc=
+github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7/go.mod h1:V3Y4KbttaCwyg1gwkP7iai8CbQx4mZUGjd3h9GZWLKE=
 github.com/filecoin-project/go-ds-versioning v0.1.2 h1:to4pTadv3IeV1wvgbCbN6Vqd+fu+7tveXgv/rCEZy6w=
 github.com/filecoin-project/go-ds-versioning v0.1.2/go.mod h1:C9/l9PnB1+mwPa26BBVpCjG/XQCB0yj/q5CK2J8X1I4=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20201016201715-d41df56b4f6a/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
@@ -321,8 +321,8 @@ github.com/filecoin-project/go-fil-commcid v0.1.0 h1:3R4ds1A9r6cr8mvZBfMYxTS88Oq
 github.com/filecoin-project/go-fil-commcid v0.1.0/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0 h1:imrrpZWEHRnNqqv0tN7LXep5bFEVOVmQWHJvl2mgsGo=
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0/go.mod h1:73S8WSEWh9vr0fDJVnKADhfIv/d6dCbAGaAGWbdJEI8=
-github.com/filecoin-project/go-fil-markets v1.27.0-rc1 h1:SYXKFONg6IaQlmXEqAEaPtb/xX57dZYMkx3LwjsLr7w=
-github.com/filecoin-project/go-fil-markets v1.27.0-rc1/go.mod h1:JkW4rU0+LqfO/DLi/wyR58AqxaUdnlSofWeu8un2XLU=
+github.com/filecoin-project/go-fil-markets v1.28.2 h1:Ev9o8BYow+lo97Bwc6oOmZ2OxdiHeIDCQsfF/w/Vldc=
+github.com/filecoin-project/go-fil-markets v1.28.2/go.mod h1:qy9LNu9t77I184VB6Pa4WKRtGfB8Vl0t8zfOLHkDqWY=
 github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
 github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
 github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0 h1:b3UDemBYN2HNfk3KOXNuxgTTxlWi3xVvbQP0IT38fvM=
@@ -330,8 +330,8 @@ github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0/go.mod h1:7aWZdaQ1b16BVoQUYR+
 github.com/filecoin-project/go-hamt-ipld/v3 v3.0.1/go.mod h1:gXpNmr3oQx8l3o7qkGyDjJjYSRX7hp/FGOStdqrWyDI=
 github.com/filecoin-project/go-hamt-ipld/v3 v3.1.0 h1:rVVNq0x6RGQIzCo1iiJlGFm9AGIZzeifggxtKMU7zmI=
 github.com/filecoin-project/go-hamt-ipld/v3 v3.1.0/go.mod h1:bxmzgT8tmeVQA1/gvBwFmYdT8SOFUwB3ovSUfG1Ux0g=
-github.com/filecoin-project/go-jsonrpc v0.2.3 h1:xdixRQfLbD8gt56dpBNKiya9gvI/79nNM13IHKTfm5E=
-github.com/filecoin-project/go-jsonrpc v0.2.3/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM=
+github.com/filecoin-project/go-jsonrpc v0.3.1 h1:qwvAUc5VwAkooquKJmfz9R2+F8znhiqcNHYjEp/NM10=
+github.com/filecoin-project/go-jsonrpc v0.3.1/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM=
 github.com/filecoin-project/go-padreader v0.0.0-20200903213702-ed5fae088b20/go.mod h1:mPn+LRRd5gEKNAtc+r3ScpW2JRU/pj4NBKdADYWHiak=
 github.com/filecoin-project/go-padreader v0.0.1 h1:8h2tVy5HpoNbr2gBRr+WD6zV6VD6XHig+ynSGJg8ZOs=
 github.com/filecoin-project/go-padreader v0.0.1/go.mod h1:VYVPJqwpsfmtoHnAmPx6MUwmrK6HIcDqZJiuZhtmfLQ=
@@ -738,8 +738,8 @@ github.com/ipfs/go-filestore v1.2.0/go.mod h1:HLJrCxRXquTeEEpde4lTLMaE/MYJZD7WHL
 github.com/ipfs/go-fs-lock v0.0.6/go.mod h1:OTR+Rj9sHiRubJh3dRhD15Juhd/+w6VPOY28L7zESmM=
 github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U=
 github.com/ipfs/go-fs-lock v0.0.7/go.mod h1:Js8ka+FNYmgQRLrRXzU3CB/+Csr1BwrRilEcvYrHhhc=
-github.com/ipfs/go-graphsync v0.14.3 h1:IXH9S7AraMQ0J6Fzcl8rqSPqLn+es33bD8OW2KNyU/o=
-github.com/ipfs/go-graphsync v0.14.3/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1taIQX/aHbBQo=
+github.com/ipfs/go-graphsync v0.14.6 h1:NPxvuUy4Z08Mg8dwpBzwgbv/PGLIufSJ1sle6iAX8yo=
+github.com/ipfs/go-graphsync v0.14.6/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1taIQX/aHbBQo=
 github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
 github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=
 github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE=
@@ -885,10 +885,10 @@ github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20211210234204-ce2a1c70cd
 github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20230102063945-1a409dc236dd h1:gMlw/MhNr2Wtp5RwGdsW23cs+yCuj9k2ON7i9MiJlRo=
 github.com/ipld/go-ipld-selector-text-lite v0.0.1 h1:lNqFsQpBHc3p5xHob2KvEg/iM5dIFn6iw4L/Hh+kS1Y=
 github.com/ipld/go-ipld-selector-text-lite v0.0.1/go.mod h1:U2CQmFb+uWzfIEF3I1arrDa5rwtj00PrpiwwCO+k1RM=
-github.com/ipni/index-provider v0.11.0 h1:q2PdK6JpYB9bzlntfkRYNBjhg4Qtko5+iXRonO88TAg=
-github.com/ipni/index-provider v0.11.0/go.mod h1:gB/wN4Mdz4MzikQubjyRRV97iS5BkD4FKB0U/bF/dY4=
-github.com/ipni/storetheindex v0.5.10 h1:r97jIZsXPuwQvePJQuStu2a/kn+Zn8X4MAdA0rU2Pu4=
-github.com/ipni/storetheindex v0.5.10/go.mod h1:SJKFCnSx4X/4ekQuZvq8pVU/7tmxkEv632Qmgu3m2bQ=
+github.com/ipni/go-libipni v0.0.8 h1:0wLfZRSBG84swmZwmaLKul/iB/FlBkkl9ZcR1ub+Z+w=
+github.com/ipni/go-libipni v0.0.8/go.mod h1:paYP9U4N3/vOzGCuN9kU972vtvw9JUcQjOKyiCFGwRk=
+github.com/ipni/index-provider v0.12.0 h1:R3F6dxxKNv4XkE4GJZNLOG0bDEbBQ/S5iztXwSD8jhQ=
+github.com/ipni/index-provider v0.12.0/go.mod h1:GhyrADJp7n06fqoc1djzkvL4buZYHzV8SoWrlxEo5F4=
 github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52 h1:QG4CGBqCeuBo6aZlGAamSkxWdgWfZGeE49eUOWJPA4c=
 github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52/go.mod h1:fdg+/X9Gg4AsAIzWpEHwnqd+QY3b7lajxyjE1m4hkq4=
 github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA=
@@ -1325,8 +1325,8 @@ github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6o
 github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ=
 github.com/multiformats/go-multicodec v0.3.1-0.20210902112759-1539a079fd61/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ=
 github.com/multiformats/go-multicodec v0.6.0/go.mod h1:GUC8upxSBE4oG+q3kWZRw/+6yC1BqO550bjhWsJbZlw=
-github.com/multiformats/go-multicodec v0.8.1 h1:ycepHwavHafh3grIbR1jIXnKCsFm0fqsfEOsJ8NtKE8=
-github.com/multiformats/go-multicodec v0.8.1/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
+github.com/multiformats/go-multicodec v0.9.0 h1:pb/dlPnzee/Sxv/j4PmkDRxCOi3hXTz3IbPKOXWJkmg=
+github.com/multiformats/go-multicodec v0.9.0/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
 github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
 github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po=
 github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
@@ -1630,8 +1630,8 @@ github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
 github.com/urfave/cli v1.22.2/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
 github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
 github.com/urfave/cli/v2 v2.0.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
-github.com/urfave/cli/v2 v2.16.3 h1:gHoFIwpPjoyIMbJp/VFd+/vuD0dAgFK4B6DpEMFJfQk=
-github.com/urfave/cli/v2 v2.16.3/go.mod h1:1CNUng3PtjQMtRzJO4FMXBQvkGtuYRxxiR9xMa7jMwI=
+github.com/urfave/cli/v2 v2.25.5 h1:d0NIAyhh5shGscroL7ek/Ya9QYQE0KNabJgiUinIQkc=
+github.com/urfave/cli/v2 v2.25.5/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasttemplate v1.0.1 h1:tY9CJiPnMXf1ERmG2EyK7gNUd+c6RKGD0IfU8WdUSz8=

itests/deals_partial_retrieval_dm-level_test.go

@@ -48,9 +48,9 @@ func TestDMLevelPartialRetrieval(t *testing.T) {
 	ctx := context.Background()
 
 	kit.QuietMiningLogs()
-	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC(), kit.MockProofs())
+	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC())
 	dh := kit.NewDealHarness(t, client, miner, miner)
-	ens.InterconnectAll().BeginMining(50 * time.Millisecond)
+	ens.InterconnectAll().BeginMiningMustPost(50 * time.Millisecond)
 
 	_, err := client.ClientImport(ctx, api.FileRef{Path: adlFixtureCar, IsCAR: true})
 	require.NoError(t, err)

itests/eth_balance_test.go

@@ -121,15 +121,24 @@ func TestEthBalanceCorrectLookup(t *testing.T) {
 	require.NoError(t, err)
 	require.True(t, ml.Receipt.ExitCode.IsSuccess())
 
-	bal, err := client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height-2), 10))
+	execTs, err := client.ChainGetTipSet(ctx, ml.TipSet)
+	require.NoError(t, err)
+
+	inclTs, err := client.ChainGetTipSet(ctx, execTs.Parents())
+	require.NoError(t, err)
+
+	inclTsParents, err := client.ChainGetTipSet(ctx, inclTs.Parents())
+	require.NoError(t, err)
+
+	bal, err := client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(inclTsParents.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, int64(0), bal.Int64())
 
-	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height-1), 10))
+	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(inclTs.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, val, bal.Int64())
 
-	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height), 10))
+	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(execTs.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, val, bal.Int64())
 }

itests/eth_deploy_test.go

@@ -29,18 +29,13 @@ import (
 // TestDeployment smoke tests the deployment of a contract via the
 // Ethereum JSON-RPC endpoint, from an EEOA.
 func TestDeployment(t *testing.T) {
-	// TODO::FVM @raulk the contract installation and invocation can be lifted into utility methods
-	// He who writes the second test, shall do that.
-	// kit.QuietMiningLogs()
-
-	// reasonable blocktime so that the tx sits in the mpool for a bit during the test.
-	// although this is non-deterministic...
-	blockTime := 1 * time.Second
+	blockTime := 100 * time.Millisecond
 	client, _, ens := kit.EnsembleMinimal(
 		t,
 		kit.MockProofs(),
 		kit.ThroughRPC())
-	ens.InterconnectAll().BeginMining(blockTime)
+
+	miners := ens.InterconnectAll().BeginMining(blockTime)
 
 	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
 	defer cancel()
@@ -93,6 +88,11 @@ func TestDeployment(t *testing.T) {
 	pendingFilter, err := client.EthNewPendingTransactionFilter(ctx)
 	require.NoError(t, err)
 
+	// Pause so we can test that everything works while the message is in the message pool.
+	for _, miner := range miners {
+		miner.Pause()
+	}
+
 	hash := client.EVM().SubmitTransaction(ctx, &tx)
 
 	mpoolTx, err := client.EthGetTransactionByHash(ctx, &hash)
@@ -103,31 +103,40 @@ func TestDeployment(t *testing.T) {
 	require.Equal(t, hash, mpoolTx.Hash)
 
 	// these fields should be nil because the tx hasn't landed on chain.
-	// TODO::FVM @raulk We can either skip the assertion if the msg has already
-	//  landed, or pause mining between the embryo creation and this assertion.
 	require.Nil(t, mpoolTx.BlockNumber)
 	require.Nil(t, mpoolTx.BlockHash)
 	require.Nil(t, mpoolTx.TransactionIndex)
 
+	// We should be able to get the message CID immediately.
+	mCid, err := client.EthGetMessageCidByTransactionHash(ctx, &hash)
+	require.NoError(t, err)
+	require.NotNil(t, mCid)
+
+	// ... and it should map back to the transaction hash.
+	mHash, err := client.EthGetTransactionHashByCid(ctx, *mCid)
+	require.NoError(t, err)
+	require.NotNil(t, mHash)
+	require.Equal(t, hash, *mHash)
+
 	changes, err := client.EthGetFilterChanges(ctx, pendingFilter)
 	require.NoError(t, err)
 	require.Len(t, changes.Results, 1)
 	require.Equal(t, hash.String(), changes.Results[0])
 
-	var receipt *api.EthTxReceipt
-	for i := 0; i < 20; i++ {
-		// TODO::FVM @raulk The right time to exit this loop isn't after
-		//  20 iterations, but when StateWaitMsg returns -- let's wait on that
-		//  event while continuing to make this assertion
-		receipt, err = client.EthGetTransactionReceipt(ctx, hash)
-		if err != nil || receipt == nil {
-			time.Sleep(500 * time.Millisecond)
-			continue
-		}
-		break
+	// Unpause mining.
+	for _, miner := range miners {
+		miner.Restart()
 	}
+
+	// Wait for the message to land.
+	_, err = client.StateWaitMsg(ctx, *mCid, 3, api.LookbackNoLimit, false)
+	require.NoError(t, err)
+
+	// Then lookup the receipt.
+	receipt, err := client.EthGetTransactionReceipt(ctx, hash)
 	require.NoError(t, err)
 	require.NotNil(t, receipt)
+
 	// logs must be an empty array, not a nil value, to avoid tooling compatibility issues
 	require.Empty(t, receipt.Logs)
 	// a correctly formed logs bloom, albeit empty, has 256 zeroes
@@ -173,21 +182,16 @@ func TestDeployment(t *testing.T) {
 	require.Nil(t, err)
 	require.True(t, reflect.DeepEqual(block1, block2))
 
-	// should be able to get the block using latest as well
-	block3, err := client.EthGetBlockByNumber(ctx, "latest", false)
-	require.Nil(t, err)
-	require.True(t, reflect.DeepEqual(block2, block3))
-
 	// verify that the block contains full tx objects
-	block4, err := client.EthGetBlockByHash(ctx, *chainTx.BlockHash, true)
+	block3, err := client.EthGetBlockByHash(ctx, *chainTx.BlockHash, true)
 	require.Nil(t, err)
-	require.Equal(t, block4.Hash, *chainTx.BlockHash)
-	require.Equal(t, block4.Number, *chainTx.BlockNumber)
+	require.Equal(t, block3.Hash, *chainTx.BlockHash)
+	require.Equal(t, block3.Number, *chainTx.BlockNumber)
 
 	// the call went through json-rpc and the response was unmarshaled
 	// into map[string]interface{}, so it has to be converted into ethtypes.EthTx
 	var foundTx *ethtypes.EthTx
-	for _, obj := range block4.Transactions {
+	for _, obj := range block3.Transactions {
 		j, err := json.Marshal(obj)
 		require.Nil(t, err)
 
@@ -202,10 +206,10 @@ func TestDeployment(t *testing.T) {
 	require.NotNil(t, foundTx)
 	require.True(t, reflect.DeepEqual(*foundTx, *chainTx))
 
-	// make sure the block got from EthGetBlockByNumber is the same
-	block5, err := client.EthGetBlockByNumber(ctx, blkNum, true)
+	// make sure the _full_ block got from EthGetBlockByNumber is the same
+	block4, err := client.EthGetBlockByNumber(ctx, blkNum, true)
 	require.Nil(t, err)
-	require.True(t, reflect.DeepEqual(block4, block5))
+	require.True(t, reflect.DeepEqual(block3, block4))
 
 	// Verify that the deployer is now an account.
 	client.AssertActorType(ctx, deployer, manifest.EthAccountKey)

itests/eth_fee_history_test.go

@@ -50,19 +50,6 @@ func TestEthFeeHistory(t *testing.T) {
 	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
 	defer cancel()
 
-	heads, err := client.ChainNotify(ctx)
-	require.NoError(err)
-
-	// Save the full view of the tipsets to calculate the answer when there are null rounds
-	tsHeights := []int{1}
-	go func() {
-		for chg := range heads {
-			for _, c := range chg {
-				tsHeights = append(tsHeights, int(c.Val.Height()))
-			}
-		}
-	}()
-
 	miner := ens.InterconnectAll().BeginMining(blockTime)
 
 	client.WaitTillChain(ctx, kit.HeightAtLeast(7))
@@ -89,6 +76,16 @@ func TestEthFeeHistory(t *testing.T) {
 		}
 	}()
 
+	currTs, err := client.ChainHead(ctx)
+	require.NoError(err)
+
+	var tsHeights []int
+	for currTs.Height() != 0 {
+		tsHeights = append(tsHeights, int(currTs.Height()))
+		currTs, err = client.ChainGetTipSet(ctx, currTs.Parents())
+		require.NoError(err)
+	}
+
 	sort.Ints(tsHeights)
 
 	// because of the deferred execution, the last tipset is not executed yet,

itests/eth_filter_test.go

@@ -425,6 +425,11 @@ func TestEthNewFilterDefaultSpec(t *testing.T) {
 	elogs, err := parseEthLogsFromFilterResult(res)
 	require.NoError(err)
 	AssertEthLogs(t, elogs, expected, received)
+
+	// giving a number to fromBlock/toBlock needs to be in hex format, so make sure passing in decimal fails
+	blockNrInDecimal := "2812200"
+	_, err = client.EthNewFilter(ctx, &ethtypes.EthFilterSpec{FromBlock: &blockNrInDecimal, ToBlock: &blockNrInDecimal})
+	require.Error(err, "expected error when fromBlock/toBlock is not in hex format")
 }
 
 func TestEthGetLogsBasic(t *testing.T) {

itests/kit/blockminer.go

@@ -184,7 +184,8 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 
 			dlinfo, err := bm.miner.FullNode.StateMinerProvingDeadline(ctx, bm.miner.ActorAddr, ts.Key())
 			require.NoError(bm.t, err)
-			if ts.Height()+1+abi.ChainEpoch(nulls) >= dlinfo.Last() { // Next block brings us past the last epoch in dline, we need to wait for miner to post
+			if ts.Height()+5+abi.ChainEpoch(nulls) >= dlinfo.Last() { // Next block brings us past the last epoch in dline, we need to wait for miner to post
+				bm.t.Logf("forcing post to get in before deadline closes at %d", dlinfo.Last())
 				bm.forcePoSt(ctx, ts, dlinfo)
 			}
 
@@ -216,7 +217,8 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 				}
 				if !success {
 					// if we are mining a new null block and it brings us past deadline boundary we need to wait for miner to post
-					if ts.Height()+1+abi.ChainEpoch(nulls+i) >= dlinfo.Last() {
+					if ts.Height()+5+abi.ChainEpoch(nulls+i) >= dlinfo.Last() {
+						bm.t.Logf("forcing post to get in before deadline closes at %d", dlinfo.Last())
 						bm.forcePoSt(ctx, ts, dlinfo)
 					}
 				}
@@ -236,7 +238,7 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 					break
 				}
 
-				require.NotEqual(bm.t, i, nloops-1, "block never managed to sync to node")
+				require.NotEqual(bm.t, i, nloops-1, "block at height %d never managed to sync to node, which is at height %d", target, ts.Height())
 				time.Sleep(time.Millisecond * 10)
 			}
 
@@ -348,7 +350,7 @@ func (bm *BlockMiner) MineUntilBlock(ctx context.Context, fn *TestFullNode, cb f
 					break
 				}
 
-				require.NotEqual(bm.t, i, nloops-1, "block never managed to sync to node")
+				require.NotEqual(bm.t, i, nloops-1, "block at height %d never managed to sync to node, which is at height %d", epoch, ts.Height())
 				time.Sleep(time.Millisecond * 10)
 			}
 

itests/kit/client.go

@@ -80,7 +80,7 @@ func RunClientTest(t *testing.T, cmds []*lcli.Command, clientNode *TestFullNode)
 	dealStatus := ""
 	for {
 		// client list-deals
-		out = clientCLI.RunCmd("client", "list-deals")
+		out = clientCLI.RunCmd("client", "list-deals", "--show-failed")
 		fmt.Println("list-deals:\n", out)
 
 		lines := strings.Split(out, "\n")

itests/net_test.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"fmt"
 	"testing"
+	"time"
 
 	"github.com/libp2p/go-libp2p/core/network"
 	"github.com/libp2p/go-libp2p/core/peer"
@@ -220,14 +221,17 @@ func TestNetBlockIPAddr(t *testing.T) {
 	firstAddrInfo, _ := firstNode.NetAddrsListen(ctx)
 	secondAddrInfo, _ := secondNode.NetAddrsListen(ctx)
 
-	var secondNodeIPs []string
-
+	secondNodeIPsMap := map[string]struct{}{} // so we can deduplicate
 	for _, addr := range secondAddrInfo.Addrs {
 		ip, err := manet.ToIP(addr)
 		if err != nil {
 			continue
 		}
-		secondNodeIPs = append(secondNodeIPs, ip.String())
+		secondNodeIPsMap[ip.String()] = struct{}{}
+	}
+	var secondNodeIPs []string
+	for s := range secondNodeIPsMap {
+		secondNodeIPs = append(secondNodeIPs, s)
 	}
 
 	// Sanity check that we're not already connected somehow
@@ -243,6 +247,8 @@ func TestNetBlockIPAddr(t *testing.T) {
 	list, err := firstNode.NetBlockList(ctx)
 	require.NoError(t, err)
 
+	fmt.Println(list.IPAddrs)
+	fmt.Println(secondNodeIPs)
 	require.Equal(t, len(list.IPAddrs), len(secondNodeIPs), "expected %d blocked IPs", len(secondNodeIPs))
 	for _, blockedIP := range list.IPAddrs {
 		found := false
@@ -256,10 +262,14 @@ func TestNetBlockIPAddr(t *testing.T) {
 		require.True(t, found, "blocked IP %s is not one of secondNodeIPs", blockedIP)
 	}
 
-	require.Error(t, secondNode.NetConnect(ctx, firstAddrInfo), "shouldn't be able to connect to second node")
-	connectedness, err = secondNode.NetConnectedness(ctx, firstAddrInfo.ID)
-	require.NoError(t, err, "failed to determine connectedness")
-	require.NotEqual(t, connectedness, network.Connected)
+	// a QUIC connection might still succeed when gated, but will be killed right after the handshake
+	_ = secondNode.NetConnect(ctx, firstAddrInfo)
+
+	require.Eventually(t, func() bool {
+		connectedness, err = secondNode.NetConnectedness(ctx, firstAddrInfo.ID)
+		require.NoError(t, err, "failed to determine connectedness")
+		return connectedness != network.Connected
+	}, time.Second*5, time.Millisecond*10)
 
 	// stm: @NETWORK_COMMON_BLOCK_REMOVE_001
 	err = firstNode.NetBlockRemove(ctx, api.NetBlockList{IPAddrs: secondNodeIPs})

itests/paych_api_test.go

@@ -10,7 +10,6 @@ import (
 	cbor "github.com/ipfs/go-ipld-cbor"
 	"github.com/stretchr/testify/require"
 
-	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/go-state-types/big"
 	paychtypes "github.com/filecoin-project/go-state-types/builtin/v8/paych"
@@ -19,7 +18,6 @@ import (
 	"github.com/filecoin-project/lotus/blockstore"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/actors/adt"
-	"github.com/filecoin-project/lotus/chain/actors/builtin"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/paych"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
 	"github.com/filecoin-project/lotus/chain/events"
@@ -52,8 +50,7 @@ func TestPaymentChannelsAPI(t *testing.T) {
 		Miner(&miner, &paymentCreator, kit.WithAllSubsystems()).
 		Start().
 		InterconnectAll()
-	bms := ens.BeginMiningMustPost(blockTime)
-	bm := bms[0]
+	ens.BeginMiningMustPost(blockTime)
 
 	waitRecvInSync := func() {
 		// paymentCreator is the block miner, in some cases paymentReceiver may fall behind, so we wait for it to catch up
@@ -197,7 +194,12 @@ func TestPaymentChannelsAPI(t *testing.T) {
 	require.Errorf(t, err, "Expected shortfall error of %d", excessAmt)
 
 	// wait for the settlement period to pass before collecting
-	waitForBlocks(ctx, t, bm, paymentReceiver, receiverAddr, policy.PaychSettleDelay)
+	head, err := paymentReceiver.ChainHead(ctx)
+	require.NoError(t, err)
+
+	settleHeight := head.Height() + policy.PaychSettleDelay + 5
+	paymentReceiver.WaitTillChain(ctx, kit.HeightAtLeast(settleHeight))
+	paymentCreator.WaitTillChain(ctx, kit.HeightAtLeast(settleHeight))
 
 	creatorPreCollectBalance, err := paymentCreator.WalletBalance(ctx, createrAddr)
 	require.NoError(t, err)
@@ -226,31 +228,6 @@ func TestPaymentChannelsAPI(t *testing.T) {
 	require.EqualValues(t, abi.NewTokenAmount(expectedRefund), delta, "did not send correct funds from creator: expected %d, got %d", expectedRefund, delta)
 }
 
-func waitForBlocks(ctx context.Context, t *testing.T, bm *kit.BlockMiner, paymentReceiver kit.TestFullNode, receiverAddr address.Address, count int) {
-	// We need to add null blocks in batches, if we add too many the chain can't sync
-	batchSize := 60
-	for i := 0; i < count; i += batchSize {
-		size := batchSize
-		if i > count {
-			size = count - i
-		}
-
-		// Add a batch of null blocks to advance the chain quicker through finalities.
-		bm.InjectNulls(abi.ChainEpoch(size - 1))
-
-		// Add a real block
-		m, err := paymentReceiver.MpoolPushMessage(ctx, &types.Message{
-			To:    builtin.BurntFundsActorAddr,
-			From:  receiverAddr,
-			Value: types.NewInt(0),
-		}, nil)
-		require.NoError(t, err)
-
-		_, err = paymentReceiver.StateWaitMsg(ctx, m.Cid(), 1, api.LookbackNoLimit, true)
-		require.NoError(t, err)
-	}
-}
-
 func waitForMessage(ctx context.Context, t *testing.T, paymentCreator kit.TestFullNode, msgCid cid.Cid, duration time.Duration, desc string) *api.MsgLookup {
 	ctx, cancel := context.WithTimeout(ctx, duration)
 	defer cancel()

itests/sector_finalize_early_test.go

@@ -31,8 +31,8 @@ func TestDealsWithFinalizeEarly(t *testing.T) {
 	var blockTime = 50 * time.Millisecond
 
 	// We use two miners so that in case the actively tested miner misses PoSt, we still have a blockchain
-	client, miner, _, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC(), kit.MutateSealingConfig(func(sc *config.SealingConfig) { sc.FinalizeEarly = true })) // no mock proofs.
-	ens.InterconnectAll().BeginMining(blockTime)
+	client, miner, poster, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC(), kit.MutateSealingConfig(func(sc *config.SealingConfig) { sc.FinalizeEarly = true })) // no mock proofs.
+	ens.InterconnectAll().BeginMiningMustPost(blockTime, poster)
 	dh := kit.NewDealHarness(t, client, miner, miner)
 
 	ctx := context.Background()

itests/sector_import_simple_test.go

@@ -39,7 +39,9 @@ func TestSectorImportAfterPC2(t *testing.T) {
 	////////
 	// Start a miner node
 
-	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC())
+	// We use two miners so that in case the actively tested miner misses PoSt, we still have a blockchain
+	client, miner, _, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC())
+
 	ens.InterconnectAll().BeginMining(blockTime)
 
 	ctx := context.Background()

itests/worker_test.go

@@ -1,6 +1,7 @@
 package itests
 
 import (
+	"bytes"
 	"context"
 	"strings"
 	"sync/atomic"
@@ -13,13 +14,17 @@ import (
 
 	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
+	miner11 "github.com/filecoin-project/go-state-types/builtin/v11/miner"
+	"github.com/filecoin-project/go-state-types/network"
 
 	"github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
 	"github.com/filecoin-project/lotus/itests/kit"
 	"github.com/filecoin-project/lotus/node"
+	"github.com/filecoin-project/lotus/node/config"
 	"github.com/filecoin-project/lotus/node/impl"
+	"github.com/filecoin-project/lotus/node/modules"
 	"github.com/filecoin-project/lotus/node/repo"
 	"github.com/filecoin-project/lotus/storage/paths"
 	sealing "github.com/filecoin-project/lotus/storage/pipeline"
@@ -500,3 +505,228 @@ func TestWorkerName(t *testing.T) {
 
 	require.True(t, found)
 }
+
+// Tests that V1_1 proofs on post workers with faults
+func TestWindowPostV1P1NV20WorkerFault(t *testing.T) {
+	kit.QuietMiningLogs()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	blocktime := 2 * time.Millisecond
+
+	sectors := 2 * 48 * 2
+	var badsector uint64 = 100000
+
+	client, miner, _, ens := kit.EnsembleWorker(t,
+		kit.PresealSectors(sectors), // 2 sectors per partition, 2 partitions in all 48 deadlines
+		kit.GenesisNetworkVersion(network.Version20),
+		kit.ConstructorOpts(
+			node.Override(new(config.ProvingConfig), func() config.ProvingConfig {
+				c := config.DefaultStorageMiner()
+				c.Proving.DisableBuiltinWindowPoSt = true
+				return c.Proving
+			}),
+			node.Override(new(*wdpost.WindowPoStScheduler), modules.WindowPostScheduler(
+				config.DefaultStorageMiner().Fees,
+				config.ProvingConfig{
+					DisableBuiltinWindowPoSt:  true,
+					DisableBuiltinWinningPoSt: false,
+					DisableWDPoStPreChecks:    false,
+				},
+			)),
+			node.Override(new(paths.Store), func(store *paths.Remote) paths.Store {
+				return &badWorkerStorage{
+					Store:       store,
+					badsector:   &badsector,
+					notBadCount: 1,
+				}
+			})),
+		kit.ThroughRPC(),
+		kit.WithTaskTypes([]sealtasks.TaskType{sealtasks.TTGenerateWindowPoSt}),
+		kit.WithWorkerStorage(func(store paths.Store) paths.Store {
+			return &badWorkerStorage{
+				Store:     store,
+				badsector: &badsector,
+			}
+		}))
+
+	bm := ens.InterconnectAll().BeginMining(blocktime)[0]
+
+	maddr, err := miner.ActorAddress(ctx)
+	require.NoError(t, err)
+
+	// wait for sectors to be committed
+	require.Eventually(t, func() bool {
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		parts, err := client.StateMinerPartitions(ctx, maddr, di.Index, types.EmptyTSK)
+		require.NoError(t, err)
+
+		return len(parts) > 1
+	}, 30*time.Second, 100*time.Millisecond)
+
+	// Wait until just before a deadline opens
+	{
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		di = di.NextNotElapsed()
+
+		t.Log("Running one proving period")
+		waitUntil := di.Open + di.WPoStChallengeWindow - di.WPoStChallengeLookback - 1
+		client.WaitTillChain(ctx, kit.HeightAtLeast(waitUntil))
+
+		t.Log("Waiting for post message")
+		bm.Stop()
+	}
+
+	// Remove one sector in the next deadline (so it's skipped)
+	{
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		parts, err := client.StateMinerPartitions(ctx, maddr, di.Index+1, types.EmptyTSK)
+		require.NoError(t, err)
+		require.Greater(t, len(parts), 0)
+
+		secs := parts[0].AllSectors
+		n, err := secs.Count()
+		require.NoError(t, err)
+		require.Equal(t, uint64(2), n)
+
+		// Drop the sector in first partition
+		sid, err := secs.First()
+		require.NoError(t, err)
+
+		t.Logf("Drop sector %d; dl %d part %d", sid, di.Index, 0)
+
+		atomic.StoreUint64(&badsector, sid)
+		require.NoError(t, err)
+	}
+
+	bm.MineBlocksMustPost(ctx, 2*time.Millisecond)
+
+	mi, err := client.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+	require.NoError(t, err)
+
+	wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+	require.NoError(t, err)
+	en := wact.Nonce
+
+	// wait for a new message to be sent from worker address, it will be a PoSt
+
+waitForProof:
+	for {
+		//stm: @CHAIN_STATE_GET_ACTOR_001
+		wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+		require.NoError(t, err)
+		if wact.Nonce > en {
+			break waitForProof
+		}
+
+		build.Clock.Sleep(blocktime)
+	}
+
+	slm, err := client.StateListMessages(ctx, &api.MessageMatch{To: maddr}, types.EmptyTSK, 0)
+	require.NoError(t, err)
+
+	pmr, err := client.StateSearchMsg(ctx, types.EmptyTSK, slm[0], -1, false)
+	require.NoError(t, err)
+
+	nv, err := client.StateNetworkVersion(ctx, pmr.TipSet)
+	require.NoError(t, err)
+	require.Equal(t, network.Version20, nv)
+
+	require.True(t, pmr.Receipt.ExitCode.IsSuccess())
+
+	slmsg, err := client.ChainGetMessage(ctx, slm[0])
+	require.NoError(t, err)
+
+	var params miner11.SubmitWindowedPoStParams
+	require.NoError(t, params.UnmarshalCBOR(bytes.NewBuffer(slmsg.Params)))
+	require.Equal(t, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1, params.Proofs[0].PoStProof)
+
+	require.Len(t, params.Partitions, 2)
+	sc0, err := params.Partitions[0].Skipped.Count()
+	require.NoError(t, err)
+	require.Equal(t, uint64(1), sc0)
+	sc1, err := params.Partitions[1].Skipped.Count()
+	require.NoError(t, err)
+	require.Equal(t, uint64(0), sc1)
+}
+
+// Tests that V1_1 proofs on post worker
+func TestWindowPostV1P1NV20Worker(t *testing.T) {
+	kit.QuietMiningLogs()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	blocktime := 2 * time.Millisecond
+
+	client, miner, _, ens := kit.EnsembleWorker(t,
+		kit.GenesisNetworkVersion(network.Version20),
+		kit.ConstructorOpts(
+			node.Override(new(config.ProvingConfig), func() config.ProvingConfig {
+				c := config.DefaultStorageMiner()
+				c.Proving.DisableBuiltinWindowPoSt = true
+				return c.Proving
+			}),
+			node.Override(new(*wdpost.WindowPoStScheduler), modules.WindowPostScheduler(
+				config.DefaultStorageMiner().Fees,
+				config.ProvingConfig{
+					DisableBuiltinWindowPoSt:  true,
+					DisableBuiltinWinningPoSt: false,
+					DisableWDPoStPreChecks:    false,
+				},
+			))),
+		kit.ThroughRPC(),
+		kit.WithTaskTypes([]sealtasks.TaskType{sealtasks.TTGenerateWindowPoSt}))
+
+	ens.InterconnectAll().BeginMining(blocktime)
+
+	maddr, err := miner.ActorAddress(ctx)
+	require.NoError(t, err)
+
+	mi, err := client.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+	require.NoError(t, err)
+
+	wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+	require.NoError(t, err)
+	en := wact.Nonce
+
+	// wait for a new message to be sent from worker address, it will be a PoSt
+
+waitForProof:
+	for {
+		//stm: @CHAIN_STATE_GET_ACTOR_001
+		wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+		require.NoError(t, err)
+		if wact.Nonce > en {
+			break waitForProof
+		}
+
+		build.Clock.Sleep(blocktime)
+	}
+
+	slm, err := client.StateListMessages(ctx, &api.MessageMatch{To: maddr}, types.EmptyTSK, 0)
+	require.NoError(t, err)
+
+	pmr, err := client.StateSearchMsg(ctx, types.EmptyTSK, slm[0], -1, false)
+	require.NoError(t, err)
+
+	nv, err := client.StateNetworkVersion(ctx, pmr.TipSet)
+	require.NoError(t, err)
+	require.Equal(t, network.Version20, nv)
+
+	require.True(t, pmr.Receipt.ExitCode.IsSuccess())
+
+	slmsg, err := client.ChainGetMessage(ctx, slm[0])
+	require.NoError(t, err)
+
+	var params miner11.SubmitWindowedPoStParams
+	require.NoError(t, params.UnmarshalCBOR(bytes.NewBuffer(slmsg.Params)))
+	require.Equal(t, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1, params.Proofs[0].PoStProof)
+}

lib/readerutil/readerutil.go

@@ -0,0 +1,44 @@
+package readerutil
+
+import (
+	"io"
+	"os"
+)
+
+// NewReadSeekerFromReaderAt returns a new io.ReadSeeker from a io.ReaderAt.
+// The returned io.ReadSeeker will read from the io.ReaderAt starting at the
+// given base offset.
+func NewReadSeekerFromReaderAt(readerAt io.ReaderAt, base int64) io.ReadSeeker {
+	return &readSeekerFromReaderAt{
+		readerAt: readerAt,
+		base:     base,
+		pos:      0,
+	}
+}
+
+type readSeekerFromReaderAt struct {
+	readerAt io.ReaderAt
+	base     int64
+	pos      int64
+}
+
+func (rs *readSeekerFromReaderAt) Read(p []byte) (n int, err error) {
+	n, err = rs.readerAt.ReadAt(p, rs.pos+rs.base)
+	rs.pos += int64(n)
+	return n, err
+}
+
+func (rs *readSeekerFromReaderAt) Seek(offset int64, whence int) (int64, error) {
+	switch whence {
+	case io.SeekStart:
+		rs.pos = offset
+	case io.SeekCurrent:
+		rs.pos += offset
+	case io.SeekEnd:
+		return 0, io.ErrUnexpectedEOF
+	default:
+		return 0, os.ErrInvalid
+	}
+
+	return rs.pos, nil
+}

metrics/metrics.go

@@ -61,6 +61,10 @@ var (
 
 	// vm execution
 	ExecutionLane, _ = tag.NewKey("lane")
+
+	// piecereader
+	PRReadType, _ = tag.NewKey("pr_type") // seq / rand
+	PRReadSize, _ = tag.NewKey("pr_size") // small / big
 )
 
 // Measures
@@ -153,8 +157,9 @@ var (
 	SchedCycleOpenWindows                = stats.Int64("sched/assigner_cycle_open_window", "Number of open windows in scheduling cycles", stats.UnitDimensionless)
 	SchedCycleQueueSize                  = stats.Int64("sched/assigner_cycle_task_queue_entry", "Number of task queue entries in scheduling cycles", stats.UnitDimensionless)
 
-	DagStorePRInitCount        = stats.Int64("dagstore/pr_init_count", "PieceReader init count", stats.UnitDimensionless)
-	DagStorePRBytesRequested   = stats.Int64("dagstore/pr_requested_bytes", "PieceReader requested bytes", stats.UnitBytes)
+	DagStorePRInitCount      = stats.Int64("dagstore/pr_init_count", "PieceReader init count", stats.UnitDimensionless)
+	DagStorePRBytesRequested = stats.Int64("dagstore/pr_requested_bytes", "PieceReader requested bytes", stats.UnitBytes)
+
 	DagStorePRBytesDiscarded   = stats.Int64("dagstore/pr_discarded_bytes", "PieceReader discarded bytes", stats.UnitBytes)
 	DagStorePRDiscardCount     = stats.Int64("dagstore/pr_discard_count", "PieceReader discard count", stats.UnitDimensionless)
 	DagStorePRSeekBackCount    = stats.Int64("dagstore/pr_seek_back_count", "PieceReader seek back count", stats.UnitDimensionless)
@@ -162,6 +167,12 @@ var (
 	DagStorePRSeekBackBytes    = stats.Int64("dagstore/pr_seek_back_bytes", "PieceReader seek back bytes", stats.UnitBytes)
 	DagStorePRSeekForwardBytes = stats.Int64("dagstore/pr_seek_forward_bytes", "PieceReader seek forward bytes", stats.UnitBytes)
 
+	DagStorePRAtHitBytes       = stats.Int64("dagstore/pr_at_hit_bytes", "PieceReader ReadAt bytes from cache", stats.UnitBytes)
+	DagStorePRAtHitCount       = stats.Int64("dagstore/pr_at_hit_count", "PieceReader ReadAt from cache hits", stats.UnitDimensionless)
+	DagStorePRAtCacheFillCount = stats.Int64("dagstore/pr_at_cache_fill_count", "PieceReader ReadAt full cache fill count", stats.UnitDimensionless)
+	DagStorePRAtReadBytes      = stats.Int64("dagstore/pr_at_read_bytes", "PieceReader ReadAt bytes read from source", stats.UnitBytes)    // PRReadSize tag
+	DagStorePRAtReadCount      = stats.Int64("dagstore/pr_at_read_count", "PieceReader ReadAt reads from source", stats.UnitDimensionless) // PRReadSize tag
+
 	// splitstore
 	SplitstoreMiss                  = stats.Int64("splitstore/miss", "Number of misses in hotstre access", stats.UnitDimensionless)
 	SplitstoreCompactionTimeSeconds = stats.Float64("splitstore/compaction_time", "Compaction time in seconds", stats.UnitSeconds)
@@ -487,6 +498,7 @@ var (
 	DagStorePRBytesRequestedView = &view.View{
 		Measure:     DagStorePRBytesRequested,
 		Aggregation: view.Sum(),
+		TagKeys:     []tag.Key{PRReadType},
 	}
 	DagStorePRBytesDiscardedView = &view.View{
 		Measure:     DagStorePRBytesDiscarded,
@@ -513,6 +525,29 @@ var (
 		Aggregation: view.Sum(),
 	}
 
+	DagStorePRAtHitBytesView = &view.View{
+		Measure:     DagStorePRAtHitBytes,
+		Aggregation: view.Sum(),
+	}
+	DagStorePRAtHitCountView = &view.View{
+		Measure:     DagStorePRAtHitCount,
+		Aggregation: view.Count(),
+	}
+	DagStorePRAtCacheFillCountView = &view.View{
+		Measure:     DagStorePRAtCacheFillCount,
+		Aggregation: view.Count(),
+	}
+	DagStorePRAtReadBytesView = &view.View{
+		Measure:     DagStorePRAtReadBytes,
+		Aggregation: view.Sum(),
+		TagKeys:     []tag.Key{PRReadSize},
+	}
+	DagStorePRAtReadCountView = &view.View{
+		Measure:     DagStorePRAtReadCount,
+		Aggregation: view.Count(),
+		TagKeys:     []tag.Key{PRReadSize},
+	}
+
 	// splitstore
 	SplitstoreMissView = &view.View{
 		Measure:     SplitstoreMiss,
@@ -779,6 +814,11 @@ var MinerNodeViews = append([]*view.View{
 	DagStorePRSeekForwardCountView,
 	DagStorePRSeekBackBytesView,
 	DagStorePRSeekForwardBytesView,
+	DagStorePRAtHitBytesView,
+	DagStorePRAtHitCountView,
+	DagStorePRAtCacheFillCountView,
+	DagStorePRAtReadBytesView,
+	DagStorePRAtReadCountView,
 }, DefaultViews...)
 
 var GatewayNodeViews = append([]*view.View{

miner/miner.go

@@ -324,7 +324,7 @@ minerLoop:
 					"block-time", btime, "time", build.Clock.Now(), "difference", build.Clock.Since(btime))
 			}
 
-			if err := m.sf.MinedBlock(ctx, b.Header, base.TipSet.Height()+base.NullRounds); err != nil {
+			if _, err = m.sf.MinedBlock(ctx, b.Header, base.TipSet.Height()+base.NullRounds); err != nil {
 				log.Errorf("<!!> SLASH FILTER ERROR: %s", err)
 				if os.Getenv("LOTUS_MINER_NO_SLASHFILTER") != "_yes_i_know_i_can_and_probably_will_lose_all_my_fil_and_power_" {
 					continue

node/builder.go

@@ -89,6 +89,8 @@ const (
 
 	// health checks
 	CheckFDLimit
+	CheckFvmConcurrency
+	LegacyMarketsEOL
 
 	// libp2p
 	PstoreAddSelfKeysKey
@@ -164,6 +166,7 @@ func defaults() []Option {
 		Override(new(dtypes.NodeStartTime), FromVal(dtypes.NodeStartTime(time.Now()))),
 
 		Override(CheckFDLimit, modules.CheckFdLimit(build.DefaultFDLimit)),
+		Override(CheckFvmConcurrency, modules.CheckFvmConcurrency()),
 
 		Override(new(system.MemoryConstraints), modules.MemoryConstraints),
 		Override(InitMemoryWatchdog, modules.MemoryWatchdog),

node/builder_miner.go

@@ -147,6 +147,10 @@ func ConfigStorageMiner(c interface{}) Option {
 		),
 
 		If(cfg.Subsystems.EnableMarkets,
+
+			// Alert that legacy-markets is being deprecated
+			Override(LegacyMarketsEOL, modules.LegacyMarketsEOL),
+
 			// Markets
 			Override(new(dtypes.StagingBlockstore), modules.StagingBlockstore),
 			Override(new(dtypes.StagingGraphsync), modules.StagingGraphsync(cfg.Dealmaking.SimultaneousTransfersForStorage, cfg.Dealmaking.SimultaneousTransfersForStoragePerClient, cfg.Dealmaking.SimultaneousTransfersForRetrieval)),

node/config/def.go

@@ -57,6 +57,10 @@ func defCommon() Common {
 			ListenAddresses: []string{
 				"/ip4/0.0.0.0/tcp/0",
 				"/ip6/::/tcp/0",
+				"/ip4/0.0.0.0/udp/0/quic-v1",
+				"/ip6/::/udp/0/quic-v1",
+				"/ip4/0.0.0.0/udp/0/quic-v1/webtransport",
+				"/ip6/::/udp/0/quic-v1/webtransport",
 			},
 			AnnounceAddresses:   []string{},
 			NoAnnounceAddresses: []string{},

node/config/doc_gen.go

@@ -473,7 +473,8 @@ Set to 0 to keep all mappings`,
 			Name: "EnableMsgIndex",
 			Type: "bool",
 
-			Comment: `EnableMsgIndex enables indexing of messages on chain.`,
+			Comment: `EXPERIMENTAL FEATURE. USE WITH CAUTION
+EnableMsgIndex enables indexing of messages on chain.`,
 		},
 	},
 	"IndexProviderConfig": []DocField{
@@ -1106,15 +1107,7 @@ required to have expiration of at least the soonest-ending deal`,
 			Name: "MinTargetUpgradeSectorExpiration",
 			Type: "uint64",
 
-			Comment: `When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-be selected based on lowest initial pledge.
-
-Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-
-Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-initial pledge - upgrade sectors will always be chosen based on longest expiration`,
+			Comment: `DEPRECATED: Target expiration is no longer used`,
 		},
 		{
 			Name: "CommittedCapacitySectorLifetime",

node/config/types.go

@@ -352,15 +352,7 @@ type SealingConfig struct {
 	// required to have expiration of at least the soonest-ending deal
 	MinUpgradeSectorExpiration uint64
 
-	// When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-	// be selected based on lowest initial pledge.
-	//
-	// Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-	// selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-	// where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-	//
-	// Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-	// initial pledge - upgrade sectors will always be chosen based on longest expiration
+	// DEPRECATED: Target expiration is no longer used
 	MinTargetUpgradeSectorExpiration uint64
 
 	// CommittedCapacitySectorLifetime is the duration a Committed Capacity (CC) sector will
@@ -736,6 +728,7 @@ type Events struct {
 }
 
 type IndexConfig struct {
+	// EXPERIMENTAL FEATURE. USE WITH CAUTION
 	// EnableMsgIndex enables indexing of messages on chain.
 	EnableMsgIndex bool
 }