Merge pull request #1 from vulcanize/ian/get_many_blockstore

GetMany badgerdb blockstore method

.gitignore

@@ -52,3 +52,4 @@ dist/
 # The following files are checked into git and result
 # in dirty git state if removed from the docker context
 !extern/filecoin-ffi/rust/filecoin.pc
+!extern/test-vectors

CHANGELOG.md

@@ -1,178 +1,10 @@
 # Lotus changelog
 
-# v1.23.2 / 2023-06-28
-
-This is a patch release on top of 1.23.1 containing the fix for https://github.com/filecoin-project/lotus/issues/10906
-This fixes the syncing issue seen by many node operators/SPs, usually when performing actions which would result in msgs staying in their mpool for longer periods of time (ex. PSD) resulting in these msgs being republished multiple times and possibly lowering your peer scores. Please refer to the above issue for more details.
-We'd recommend everyone to accept this fix to better overall network health
-
-## Improvements
-- fix: pubsub: do not treat ErrExistingNonce as Reject
-
-
-# v1.23.1 / 2023-06-20
-
-This is an optional feature release of Lotus. This feature release includes numerous improvements and enhancements for node operators, ETH RPC-providers and storage providers.
-
-**☢️ Upgrade Warnings ☢️**
-
-If you are upgrading to this release candidate from Lotus v1.22.1, please make sure to read the upgrade warnings section in the [v1.23.0 release first.](https://github.com/filecoin-project/lotus/releases/tag/v1.23.0)
-
-- *Storage providers:* The Lotus-Miner legacy-markets has been disbled by default in this feature release and will be removed in the near term future. Users are adviced to migrate to [Boost](https://boost.filecoin.io) or other SP markets systems.
-
-## Highlights
-
-**🛣 Execution Lanes 🛣**
-This feature release introduces VM Execution Lanes! Execution lanes efficiently divide the workload between system processes  (chainsync) and RPC requests. This way syncing the chain will not be at the mercy of responding to users' requests and RPC providers nodes should have less problems catching up.
-
-To take advantage of VM Execution Lanes, you need to set up two environment variables:
-- `LOTUS_FVM_CONCURRENCY` - read more about how this value should be set to [here](https://lotus.filecoin.io/lotus/configure/ethereum-rpc/#environment-variables)
-- `LOTUS_FVM_CONCURRENCY_RESERVED = 4` 
-
-**🧱 Aggregation / Batching fixes 🔨**
-
-Numerous aggregation and batching fixes has been included in the feature release. Large `ProveCommitAggregate` and `PreCommitBatching` messages that exceeds the block limit will now automatically be split into smaller messages when sent to the chain.
-
-Additionally we have added a new feature that staggers the amount of ProveCommit messages sent simulatanously to the chain if a storage provider has been aggregating many sectors in ProveCommitAggregate message, but at the time of publishing the BaseFee is below the aggregation threshold. This stagger feature prevents issues where some of the ProveCommit messages fail with the SysErrorOutOfGas message. You can tweak how many messages will be staggered per epoch by changing `MaxSectorProveCommitsSubmittedPerEpoch` in the [sealing section of the config.toml file.](https://lotus.filecoin.io/storage-providers/advanced-configurations/sealing/#sealing-section)
-
-*NB:* While these fixes are great for the reliability of aggregation and batching on the Lotus side, it has been uncovered that aggregating ProveCommit messages for sectors containing verified deals are currently more expensive then single messages due to an issue on the actors side. We therefore do not reccomend our users to aggregate ProveCommit messages when doing verified deals until that issue has been resolved. You can follow the discussion on resolving the issue on the [actors side here.](https://github.com/filecoin-project/FIPs/discussions/689)
-
-**Unsealing CLI/API**
-
-This feature release adds a dedicated `lotus-miner sectors unseal` command and API, allowing you to unseal specific sealed sectors easily.
+# UNRELEASED
 
 ## New features
-- feat: VM Execution Lanes ([filecoin-project/lotus#10551](https://github.com/filecoin-project/lotus/pull/10551))
-   - Adds VM exections lanes, efficiently dividing the workload between system processes and RPC-requests.
-- Add API and CLI to unseal sector (#10626) ([filecoin-project/lotus#10626](https://github.com/filecoin-project/lotus/pull/10626))
-   - Adds `lotus-miner sectors unseal` cmd, and a API-method to unseal a sector.
-- feat: sealing: Split PCA/PCB batches if gas used exceeds block limit ([filecoin-project/lotus#10647](https://github.com/filecoin-project/lotus/pull/10647))
-   - Splits ProveCommitAggregate and PreCommitBatch messages into multiple messages if the message exceeds the block limit.
-- Add feature to stagger sector prove commit submission (#10543) ([filecoin-project/lotus#10543](https://github.com/filecoin-project/lotus/pull/10543))
-   - Staggers the amount of ProveCommit messages sent simultanously if a storage provider has been aggregating many message, but at the moment of publishing the BaseFee is below the threshold for aggregation to prevent unwanted SysErrorOutOfGas issues.
-- Set default for MaxSectorProveCommitsSubmittedPerEpoch ([filecoin-project/lotus#10728](https://github.com/filecoin-project/lotus/pull/10728))
-   - Sets the default amount of ProveCommits submitted per epoch to 20.
-- feat: worker: Ensure tempdir exists (#10433) ([filecoin-project/lotus#10433](https://github.com/filecoin-project/lotus/pull/10433))
-   - Ensures that a temporary directory exists on start of a lotus-worker with a custom TMPDIR set.
-- feat: sync: harden chain sync (#10756) ([filecoin-project/lotus#10756](https://github.com/filecoin-project/lotus/pull/10756))
-- feat: populate the index on snapshot import ([filecoin-project/lotus#10556](https://github.com/filecoin-project/lotus/pull/10556))
-- feat:chain: Message Index (**HIGHLY EXPERIMENTAL**) ([filecoin-project/lotus#10452](https://github.com/filecoin-project/lotus/pull/10452))
-   - MVP of a message index that allows us to accelrate StateSearchMessage and related functionality, and eventually accelerate critical chain calls (follow up).
-- feat: Add small cache to execution traces ([filecoin-project/lotus#10517](https://github.com/filecoin-project/lotus/pull/10517))
-- feat: shed: incoming block-sub chainwatch tool ([filecoin-project/lotus#10513](https://github.com/filecoin-project/lotus/pull/10513))
-
-## Improvements
-- feat: daemon: Auto-resume interrupted snapshot imports ([filecoin-project/lotus#10636](https://github.com/filecoin-project/lotus/pull/10636))
-   - Auto-resumes interrupted snapshot imports when using an URL.
-- fix: storage: Remove temp fetching files after failed fetch ([filecoin-project/lotus#10661](https://github.com/filecoin-project/lotus/pull/10661))
-   - Clean up partially fetched failed after a failed fetch on a lotus-worker.
-- feat: chainstore: batch writes of tipsets ([filecoin-project/lotus#10800](https://github.com/filecoin-project/lotus/pull/10800))
-   - Reduces the time to persist all headers from 4-5 minutes, to < 15 seconds. 
-- Check if epoch is negative in GetTipsetByHeight
-- fix: sched: Address GET_32G_MAX_CONCURRENT regression
-- fix: cli: Hide legacy markets cmds
-   - Hides the lotus-miner legacy markets commands from the lotus-miner CLI.
-- fix: ci: Debugging m1 build
-- Disable lotus markets by default (#10809) ([filecoin-project/lotus#10809](https://github.com/filecoin-project/lotus/pull/10809))
-   - Disables lotus-miner legacy markets [EOL] by default.
-- perf: mempool: lower priority optimizations (#10693) ([filecoin-project/lotus#10693](https://github.com/filecoin-project/lotus/pull/10693))
-- perf: message pool: change locks to RWMutexes for performance ([filecoin-project/lotus#10561](https://github.com/filecoin-project/lotus/pull/10561))
-- perf: eth: gas estimate set applyTsMessages false (#10546) ([filecoin-project/lotus#10546](https://github.com/filecoin-project/lotus/pull/10546))
-- Change args check ([filecoin-project/lotus#10812](https://github.com/filecoin-project/lotus/pull/10812))
-- fix: sealing: Make lotus-worker report GPU usage to miner during ReplicaUpdate task (#10806) ([filecoin-project/lotus#10806](https://github.com/filecoin-project/lotus/pull/10806))
-- fix:splitstore:Don't block when potentially holding txnLk as writer ([filecoin-project/lotus#10811](https://github.com/filecoin-project/lotus/pull/10811))
-- fix: prover: Propagate skipped sectors in local PoSt
-- fix: unseal: check if sealed/update sector exists ([filecoin-project/lotus#10639](https://github.com/filecoin-project/lotus/pull/10639))
-- fix: sealing pipeline: Allow nil message in TerminateWait ([filecoin-project/lotus#10696](https://github.com/filecoin-project/lotus/pull/10696))
-- fix: cli: Check if the sectorID exists before removing ([filecoin-project/lotus#10611](https://github.com/filecoin-project/lotus/pull/10611))
-- feat:splitstore:limit moving gc threads ([filecoin-project/lotus#10621](https://github.com/filecoin-project/lotus/pull/10621))
-- fix: cli: Make `net connect` to miner address work ([filecoin-project/lotus#10599](https://github.com/filecoin-project/lotus/pull/10599))
-- fix: log: Stop logging `file does not exists` ([filecoin-project/lotus#10588](https://github.com/filecoin-project/lotus/pull/10588))
-- Update config default value (#10605) ([filecoin-project/lotus#10605](https://github.com/filecoin-project/lotus/pull/10605))
-- fix: cap the message gas limit at the block gas limit (#10637) ([filecoin-project/lotus#10637](https://github.com/filecoin-project/lotus/pull/10637))
-- fix: miner: correctly count sector extensions ([filecoin-project/lotus#10544](https://github.com/filecoin-project/lotus/pull/10544))
-- fix:mpool: prune excess messages before selection ([filecoin-project/lotus#10648](https://github.com/filecoin-project/lotus/pull/10648))
-- fix: proving: Initialize slice with with same length as partition ([filecoin-project/lotus#10569](https://github.com/filecoin-project/lotus/pull/10569))
-- perf: Address performance of EthGetTransactionCount  ([filecoin-project/lotus#10700](https://github.com/filecoin-project/lotus/pull/10700))
-- fix: sync: reduce log from error to info ([filecoin-project/lotus#10759](https://github.com/filecoin-project/lotus/pull/10759))
-- fix: state: lotus-miner info should show deals info without admin permission ([filecoin-project/lotus#10323](https://github.com/filecoin-project/lotus/pull/10323))
-- fix: tvx: make extract-multiple support the FVM ([filecoin-project/lotus#10714](https://github.com/filecoin-project/lotus/pull/10714))
-- feat: badger: add a has check before writing to reduce duplicates ([filecoin-project/lotus#10680](https://github.com/filecoin-project/lotus/pull/10680))
-- fix: chain: record heaviest tipset before notifying (#10694) ([filecoin-project/lotus#10694](https://github.com/filecoin-project/lotus/pull/10694))
-- fix: Eth JSON-RPC api: handle messages with gasFeeCap less than baseFee (#10614) ([filecoin-project/lotus#10614](https://github.com/filecoin-project/lotus/pull/10614))
-- feat: chainstore: optimize BlockMsgsForTipset ([filecoin-project/lotus#10552](https://github.com/filecoin-project/lotus/pull/10552))
-- refactor: stop using deprecated io/ioutil ([filecoin-project/lotus#10596](https://github.com/filecoin-project/lotus/pull/10596))
-- feat: shed: refactor market cron-state command ([filecoin-project/lotus#10746](https://github.com/filecoin-project/lotus/pull/10746))
-- fix: events: don't set GC confidence to 1 ([filecoin-project/lotus#10713](https://github.com/filecoin-project/lotus/pull/10713))
-- feat: sync: validate (early) that blocks fall within range (#10691) ([filecoin-project/lotus#10691](https://github.com/filecoin-project/lotus/pull/10691))
-- chainstore: Fix raw blocks getting scanned for links during snapshots (#10684) ([filecoin-project/lotus#10684](https://github.com/filecoin-project/lotus/pull/10684))
-- fix: remove pointless panic ([filecoin-project/lotus#10690](https://github.com/filecoin-project/lotus/pull/10690))
-- fix: check for nil bcastDict (#10646) ([filecoin-project/lotus#10646](https://github.com/filecoin-project/lotus/pull/10646))
-- fix: make state compute --html work with unknown methods ([filecoin-project/lotus#10619](https://github.com/filecoin-project/lotus/pull/10619))
-- shed: get balances of evm accounts  ([filecoin-project/lotus#10489](https://github.com/filecoin-project/lotus/pull/10489))
-- feat: Use MessageIndex in WaitForMessage ([filecoin-project/lotus#10587](https://github.com/filecoin-project/lotus/pull/10587))
-- fix: searchForIndexedMsg always returns an error ([filecoin-project/lotus#10586](https://github.com/filecoin-project/lotus/pull/10586))
-- Fix: export-range: Ignore ipld Blocks not found in Receipts. ([filecoin-project/lotus#10535](https://github.com/filecoin-project/lotus/pull/10535))
-- feat: stmgr: speed up calculation of genesis circ supply ([filecoin-project/lotus#10553](https://github.com/filecoin-project/lotus/pull/10553))
-- fix: gas estimation: don't special case paych collects ([filecoin-project/lotus#10549](https://github.com/filecoin-project/lotus/pull/10549))
-- fix: tracer: emit raw peer ids for compatibility with libp2p tracer ([filecoin-project/lotus#10271](https://github.com/filecoin-project/lotus/pull/10271))
-- Merge branch 'feat/new-gw-methods'
-
-## Dependencies
-- chore: deps: update to go-libp2p 0.27.5
-- devs: update libp2p #10937
-- chore: deps: update to FVM 3.3.1 ([filecoin-project/lotus#10786](https://github.com/filecoin-project/lotus/pull/10786))
-- chore: boxo: migrate from go-libipfs to boxo ([filecoin-project/lotus#10562](https://github.com/filecoin-project/lotus/pull/10562))
-- chore: deps: update to go-state-types v0.11.0-alpha-3 ([filecoin-project/lotus#10606](https://github.com/filecoin-project/lotus/pull/10606))
-- chore: bump go-libipfs ([filecoin-project/lotus#10531](https://github.com/filecoin-project/lotus/pull/10531))
-
-## Others
-- feat:networking:  (Synchronous) Consistent Broadcast for Filecoin EC ([filecoin-project/lotus#9858](https://github.com/filecoin-project/lotus/pull/9858))
-- Revert #9858 (consistent broadcast changes) ([filecoin-project/lotus#10777](https://github.com/filecoin-project/lotus/pull/10777))
-- Update build version for release/v1.23.1
-- chore: drop flaky TestBatchDealInput subcase ([filecoin-project/lotus#10810](https://github.com/filecoin-project/lotus/pull/10810))
-- chore: changelog clean up ([filecoin-project/lotus#10744](https://github.com/filecoin-project/lotus/pull/10744))
-- chore: refactor: drop unused IsTicketWinner (#10801) ([filecoin-project/lotus#10801](https://github.com/filecoin-project/lotus/pull/10801))
-- chore: build: bump matser version to v1.23.1-dev ([filecoin-project/lotus#10709](https://github.com/filecoin-project/lotus/pull/10709))
-- fix: deflake: use 2 miners for flaky tests ([filecoin-project/lotus#10764](https://github.com/filecoin-project/lotus/pull/10764))
-- test: eth: deflake multiblock lookup test (#10769) ([filecoin-project/lotus#10769](https://github.com/filecoin-project/lotus/pull/10769))
-- shed: migrations: add reminder about continuity testing tool ([filecoin-project/lotus#10762](https://github.com/filecoin-project/lotus/pull/10762))
-- chore: merge releases into master ([filecoin-project/lotus#10742](https://github.com/filecoin-project/lotus/pull/10742))
-- test: events: fix race when recording tipsets (#10665) ([filecoin-project/lotus#10665](https://github.com/filecoin-project/lotus/pull/10665))
-- fix: build: add CBDeliveryDelay to testground ([filecoin-project/lotus#10613](https://github.com/filecoin-project/lotus/pull/10613))
-- fix: build: Fixed incorrect words that could not be compiled ([filecoin-project/lotus#10610](https://github.com/filecoin-project/lotus/pull/10610))
-- build: docker: Update GO-version ([filecoin-project/lotus#10581](https://github.com/filecoin-project/lotus/pull/10581))
-- fix: itests: Don't call t.Error in MineBlocks goroutine ([filecoin-project/lotus#10572](https://github.com/filecoin-project/lotus/pull/10572))
-- docs: api: clarify MpoolClear params ([filecoin-project/lotus#10550](https://github.com/filecoin-project/lotus/pull/10550))
-
-Contributors
-
-| Contributor | Commits | Lines ± | Files Changed |
-|-------------|---------|---------|---------------|
-| vyzo | 70 | +1990/-429 | 135 |
-| Alfonso de la Rocha | 25 | +814/-299 | 56 |
-| Steven Allen | 14 | +125/-539 | 28 |
-| Shrenuj Bansal | 13 | +482/-138 | 52 |
-| Aayush | 17 | +317/-301 | 90 |
-| Łukasz Magiera | 13 | +564/-26 | 16 |
-| Jennifer Wang | 7 | +401/-140 | 10 |
-| Fridrik Asmundsson | 14 | +315/-84 | 20 |
-| Jorropo | 2 | +139/-137 | 74 |
-| Mikers | 6 | +114/-43 | 14 |
-| Hector Sanjuan | 5 | +92/-44 | 5 |
-| Ales Dumikau | 1 | +117/-0 | 10 |
-| Mike Seiler | 4 | +51/-51 | 6 |
-| zenground0 | 6 | +33/-25 | 8 |
-| Phi | 8 | +32/-10 | 10 |
-| Aayush Rajasekaran | 1 | +1/-32 | 2 |
-| Ian Davis | 2 | +7/-10 | 3 |
-| Marcel Telka | 1 | +5/-7 | 1 |
-| ychiao | 1 | +8/-3 | 2 |
-| jennijuju | 1 | +4/-4 | 8 |
-| adlrocha | 2 | +2/-2 | 2 |
-| Jiaying Wang | 1 | +0/-4 | 1 |
-| ZenGround0 | 1 | +2/-1 | 2 |
-| Zeng Li | 1 | +1/-1 | 1 |
+- feat: Added new environment variable `LOTUS_EXEC_TRACE_CACHE_SIZE` to configure execution trace cache size ([filecoin-project/lotus#10585](https://github.com/filecoin-project/lotus/pull/10585))
+  - If unset, we default to caching 16 most recent execution traces. Node operatores may want to set this to 0 while exchanges may want to crank it up. 
 
 # v1.23.0 / 2023-04-21
 

Dockerfile.lotus

@@ -1,273 +0,0 @@
-##### DEPRECATED
-
-FROM golang:1.19.7-buster AS builder-deps
-MAINTAINER Lotus Development Team
-
-RUN apt-get update && apt-get install -y ca-certificates build-essential clang ocl-icd-opencl-dev ocl-icd-libopencl1 jq libhwloc-dev
-
-ENV XDG_CACHE_HOME="/tmp"
-
-### taken from https://github.com/rust-lang/docker-rust/blob/master/1.63.0/buster/Dockerfile
-ENV RUSTUP_HOME=/usr/local/rustup \
-    CARGO_HOME=/usr/local/cargo \
-    PATH=/usr/local/cargo/bin:$PATH \
-    RUST_VERSION=1.63.0
-
-RUN set -eux; \
-    dpkgArch="$(dpkg --print-architecture)"; \
-    case "${dpkgArch##*-}" in \
-        amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='5cc9ffd1026e82e7fb2eec2121ad71f4b0f044e88bca39207b3f6b769aaa799c' ;; \
-        arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='e189948e396d47254103a49c987e7fb0e5dd8e34b200aa4481ecc4b8e41fb929' ;; \
-        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
-    esac; \
-    url="https://static.rust-lang.org/rustup/archive/1.25.1/${rustArch}/rustup-init"; \
-    wget "$url"; \
-    echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
-    chmod +x rustup-init; \
-    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
-    rm rustup-init; \
-    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
-    rustup --version; \
-    cargo --version; \
-    rustc --version;
-### end rust
-
-FROM builder-deps AS builder-local
-MAINTAINER Lotus Development Team
-
-COPY ./ /opt/filecoin
-WORKDIR /opt/filecoin
-
-### make configurable filecoin-ffi build
-ARG FFI_BUILD_FROM_SOURCE=0
-ENV FFI_BUILD_FROM_SOURCE=${FFI_BUILD_FROM_SOURCE}
-
-RUN make clean deps
-
-
-FROM builder-local AS builder-test
-MAINTAINER Lotus Development Team
-
-WORKDIR /opt/filecoin
-
-RUN make debug
-
-
-FROM builder-local AS builder
-MAINTAINER Lotus Development Team
-
-WORKDIR /opt/filecoin
-
-ARG RUSTFLAGS=""
-ARG GOFLAGS=""
-
-RUN make lotus lotus-miner lotus-worker lotus-shed lotus-wallet lotus-gateway lotus-stats
-
-
-FROM ubuntu:20.04 AS base
-MAINTAINER Lotus Development Team
-
-# Base resources
-COPY --from=builder /etc/ssl/certs                           /etc/ssl/certs
-COPY --from=builder /lib/*/libdl.so.2         /lib/
-COPY --from=builder /lib/*/librt.so.1         /lib/
-COPY --from=builder /lib/*/libgcc_s.so.1      /lib/
-COPY --from=builder /lib/*/libutil.so.1       /lib/
-COPY --from=builder /usr/lib/*/libltdl.so.7   /lib/
-COPY --from=builder /usr/lib/*/libnuma.so.1   /lib/
-COPY --from=builder /usr/lib/*/libhwloc.so.5  /lib/
-COPY --from=builder /usr/lib/*/libOpenCL.so.1 /lib/
-
-RUN useradd -r -u 532 -U fc \
- && mkdir -p /etc/OpenCL/vendors \
- && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
-
-###
-FROM base AS lotus
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-shed /usr/local/bin/
-COPY scripts/docker-lotus-entrypoint.sh /
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_PATH /var/lib/lotus
-ENV DOCKER_LOTUS_IMPORT_SNAPSHOT https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car
-ENV DOCKER_LOTUS_IMPORT_WALLET ""
-
-RUN mkdir /var/lib/lotus /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus /var/tmp/filecoin-proof-parameters
-
-VOLUME /var/lib/lotus
-VOLUME /var/tmp/filecoin-proof-parameters
-
-USER fc
-
-EXPOSE 1234
-
-ENTRYPOINT ["/docker-lotus-entrypoint.sh"]
-
-CMD ["-help"]
-
-###
-FROM base AS lotus-wallet
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-wallet /usr/local/bin/
-
-ENV WALLET_PATH /var/lib/lotus-wallet
-
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/lib/lotus-wallet
-
-VOLUME /var/lib/lotus-wallet
-
-USER fc
-
-EXPOSE 1777
-
-ENTRYPOINT ["/usr/local/bin/lotus-wallet"]
-
-CMD ["-help"]
-
-###
-FROM base AS lotus-gateway
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-gateway /usr/local/bin/
-
-USER fc
-
-EXPOSE 1234
-
-ENTRYPOINT ["/usr/local/bin/lotus-gateway"]
-
-CMD ["-help"]
-
-
-###
-FROM base AS lotus-miner
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-miner /usr/local/bin/
-COPY scripts/docker-lotus-miner-entrypoint.sh /
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-
-RUN mkdir /var/lib/lotus-miner /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus-miner /var/tmp/filecoin-proof-parameters
-
-VOLUME /var/lib/lotus-miner
-VOLUME /var/tmp/filecoin-proof-parameters
-
-USER fc
-
-EXPOSE 2345
-
-ENTRYPOINT ["/docker-lotus-miner-entrypoint.sh"]
-
-CMD ["-help"]
-
-
-###
-FROM base AS lotus-worker
-MAINTAINER Lotus Development Team
-
-COPY --from=builder /opt/filecoin/lotus-worker /usr/local/bin/
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-
-RUN mkdir /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-worker
-
-VOLUME /var/lib/lotus-worker
-
-USER fc
-
-EXPOSE 3456
-
-ENTRYPOINT ["/usr/local/bin/lotus-worker"]
-
-CMD ["-help"]
-
-
-###
-from base as lotus-all-in-one
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-ENV LOTUS_PATH /var/lib/lotus
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-ENV WALLET_PATH /var/lib/lotus-wallet
-ENV DOCKER_LOTUS_IMPORT_SNAPSHOT https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car
-
-COPY --from=builder /opt/filecoin/lotus      /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-shed /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-wallet /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-gateway /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-miner /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-worker /usr/local/bin/
-COPY --from=builder /opt/filecoin/lotus-stats /usr/local/bin/
-
-RUN mkdir /var/tmp/filecoin-proof-parameters
-RUN mkdir /var/lib/lotus
-RUN mkdir /var/lib/lotus-miner
-RUN mkdir /var/lib/lotus-worker
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus
-RUN chown fc: /var/lib/lotus-miner
-RUN chown fc: /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-wallet
-
-
-VOLUME /var/tmp/filecoin-proof-parameters
-VOLUME /var/lib/lotus
-VOLUME /var/lib/lotus-miner
-VOLUME /var/lib/lotus-worker
-VOLUME /var/lib/lotus-wallet
-
-EXPOSE 1234
-EXPOSE 2345
-EXPOSE 3456
-EXPOSE 1777
-
-###
-from base as lotus-test
-
-ENV FILECOIN_PARAMETER_CACHE /var/tmp/filecoin-proof-parameters
-ENV LOTUS_MINER_PATH /var/lib/lotus-miner
-ENV LOTUS_PATH /var/lib/lotus
-ENV LOTUS_WORKER_PATH /var/lib/lotus-worker
-ENV WALLET_PATH /var/lib/lotus-wallet
-
-COPY --from=builder-test /opt/filecoin/lotus      /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-miner /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-worker /usr/local/bin/
-COPY --from=builder-test /opt/filecoin/lotus-seed /usr/local/bin/
-
-RUN mkdir /var/tmp/filecoin-proof-parameters
-RUN mkdir /var/lib/lotus
-RUN mkdir /var/lib/lotus-miner
-RUN mkdir /var/lib/lotus-worker
-RUN mkdir /var/lib/lotus-wallet
-RUN chown fc: /var/tmp/filecoin-proof-parameters
-RUN chown fc: /var/lib/lotus
-RUN chown fc: /var/lib/lotus-miner
-RUN chown fc: /var/lib/lotus-worker
-RUN chown fc: /var/lib/lotus-wallet
-
-
-VOLUME /var/tmp/filecoin-proof-parameters
-VOLUME /var/lib/lotus
-VOLUME /var/lib/lotus-miner
-VOLUME /var/lib/lotus-worker
-VOLUME /var/lib/lotus-wallet
-
-EXPOSE 1234
-EXPOSE 2345
-EXPOSE 3456
-EXPOSE 1777
-

Makefile

@@ -355,7 +355,7 @@ fiximports:
 	./scripts/fiximports
 
 gen: actors-code-gen type-gen cfgdoc-gen docsgen api-gen circleci fiximports
-	@echo ">>> IF YOU'VE MODIFIED THE CLI OR CONFIG, REMEMBER TO ALSO MAKE docsgen-cli"
+	@echo ">>> IF YOU'VE MODIFIED THE CLI OR CONFIG, REMEMBER TO ALSO RUN 'make docsgen-cli'"
 .PHONY: gen
 
 jen: gen

api/api_full.go

@@ -812,6 +812,7 @@ type FullNode interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error) //perm:read
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)                              //perm:read
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)                                                                               //perm:read
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)                                                                        //perm:read
 	NetVersion(ctx context.Context) (string, error)                                                                                           //perm:read
 	NetListening(ctx context.Context) (bool, error)                                                                                           //perm:read
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)                                                                       //perm:read
@@ -829,23 +830,23 @@ type FullNode interface {
 
 	// Polling method for a filter, returns event logs which occurred since last poll.
 	// (requires write perm since timestamp of last filter execution will be written)
-	EthGetFilterChanges(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:write
+	EthGetFilterChanges(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:read
 
 	// Returns event logs matching filter with given id.
 	// (requires write perm since timestamp of last filter execution will be written)
-	EthGetFilterLogs(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:write
+	EthGetFilterLogs(ctx context.Context, id ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) //perm:read
 
 	// Installs a persistent filter based on given filter spec.
-	EthNewFilter(ctx context.Context, filter *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) //perm:write
+	EthNewFilter(ctx context.Context, filter *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) //perm:read
 
 	// Installs a persistent filter to notify when a new block arrives.
-	EthNewBlockFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:write
+	EthNewBlockFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:read
 
 	// Installs a persistent filter to notify when new messages arrive in the message pool.
-	EthNewPendingTransactionFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:write
+	EthNewPendingTransactionFilter(ctx context.Context) (ethtypes.EthFilterID, error) //perm:read
 
 	// Uninstalls a filter with given id.
-	EthUninstallFilter(ctx context.Context, id ethtypes.EthFilterID) (bool, error) //perm:write
+	EthUninstallFilter(ctx context.Context, id ethtypes.EthFilterID) (bool, error) //perm:read
 
 	// Subscribe to different event types using websockets
 	// eventTypes is one or more of:
@@ -854,10 +855,10 @@ type FullNode interface {
 	//  - logs: notify new event logs that match a criteria
 	// params contains additional parameters used with the log event type
 	// The client will receive a stream of EthSubscriptionResponse values until EthUnsubscribe is called.
-	EthSubscribe(ctx context.Context, params jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) //perm:write
+	EthSubscribe(ctx context.Context, params jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) //perm:read
 
 	// Unsubscribe from a websocket subscription
-	EthUnsubscribe(ctx context.Context, id ethtypes.EthSubscriptionID) (bool, error) //perm:write
+	EthUnsubscribe(ctx context.Context, id ethtypes.EthSubscriptionID) (bool, error) //perm:read
 
 	// Returns the client version
 	Web3ClientVersion(ctx context.Context) (string, error) //perm:read

api/api_gateway.go

@@ -33,6 +33,9 @@ import (
 //  * Generate openrpc blobs
 
 type Gateway interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*MiningBaseInfo, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (MinerSectors, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*InvocResult, error)
@@ -98,6 +101,7 @@ type Gateway interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error)
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)
 	NetVersion(ctx context.Context) (string, error)
 	NetListening(ctx context.Context) (bool, error)
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)

api/docgen/docgen.go

@@ -86,6 +86,7 @@ func init() {
 	}
 
 	ExampleValues[reflect.TypeOf(addr)] = addr
+	ExampleValues[reflect.TypeOf(&addr)] = &addr
 
 	pid, err := peer.Decode("12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf")
 	if err != nil {

api/eth_aliases.go

@@ -21,6 +21,7 @@ func CreateEthRPCAliases(as apitypes.Aliaser) {
 	as.AliasMethod("eth_getStorageAt", "Filecoin.EthGetStorageAt")
 	as.AliasMethod("eth_getBalance", "Filecoin.EthGetBalance")
 	as.AliasMethod("eth_chainId", "Filecoin.EthChainId")
+	as.AliasMethod("eth_syncing", "Filecoin.EthSyncing")
 	as.AliasMethod("eth_feeHistory", "Filecoin.EthFeeHistory")
 	as.AliasMethod("eth_protocolVersion", "Filecoin.EthProtocolVersion")
 	as.AliasMethod("eth_maxPriorityFeePerGas", "Filecoin.EthMaxPriorityFeePerGas")

api/mocks/mock_full.go

@@ -1476,6 +1476,21 @@ func (mr *MockFullNodeMockRecorder) EthSubscribe(arg0, arg1 interface{}) *gomock
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EthSubscribe", reflect.TypeOf((*MockFullNode)(nil).EthSubscribe), arg0, arg1)
 }
 
+// EthSyncing mocks base method.
+func (m *MockFullNode) EthSyncing(arg0 context.Context) (ethtypes.EthSyncingResult, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "EthSyncing", arg0)
+	ret0, _ := ret[0].(ethtypes.EthSyncingResult)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// EthSyncing indicates an expected call of EthSyncing.
+func (mr *MockFullNodeMockRecorder) EthSyncing(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EthSyncing", reflect.TypeOf((*MockFullNode)(nil).EthSyncing), arg0)
+}
+
 // EthUninstallFilter mocks base method.
 func (m *MockFullNode) EthUninstallFilter(arg0 context.Context, arg1 ethtypes.EthFilterID) (bool, error) {
 	m.ctrl.T.Helper()

api/proxy_gen.go

@@ -274,9 +274,9 @@ type FullNodeMethods struct {
 
 	EthGetCode func(p0 context.Context, p1 ethtypes.EthAddress, p2 string) (ethtypes.EthBytes, error) `perm:"read"`
 
-	EthGetFilterChanges func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"write"`
+	EthGetFilterChanges func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
-	EthGetFilterLogs func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"write"`
+	EthGetFilterLogs func(p0 context.Context, p1 ethtypes.EthFilterID) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
 	EthGetLogs func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (*ethtypes.EthFilterResult, error) `perm:"read"`
 
@@ -302,21 +302,23 @@ type FullNodeMethods struct {
 
 	EthMaxPriorityFeePerGas func(p0 context.Context) (ethtypes.EthBigInt, error) `perm:"read"`
 
-	EthNewBlockFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewBlockFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"read"`
 
-	EthNewFilter func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewFilter func(p0 context.Context, p1 *ethtypes.EthFilterSpec) (ethtypes.EthFilterID, error) `perm:"read"`
 
-	EthNewPendingTransactionFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"write"`
+	EthNewPendingTransactionFilter func(p0 context.Context) (ethtypes.EthFilterID, error) `perm:"read"`
 
 	EthProtocolVersion func(p0 context.Context) (ethtypes.EthUint64, error) `perm:"read"`
 
 	EthSendRawTransaction func(p0 context.Context, p1 ethtypes.EthBytes) (ethtypes.EthHash, error) `perm:"read"`
 
-	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) `perm:"write"`
+	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) `perm:"read"`
 
-	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) `perm:"write"`
+	EthSyncing func(p0 context.Context) (ethtypes.EthSyncingResult, error) `perm:"read"`
 
-	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) `perm:"write"`
+	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) `perm:"read"`
+
+	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) `perm:"read"`
 
 	FilecoinAddressToEthAddress func(p0 context.Context, p1 address.Address) (ethtypes.EthAddress, error) `perm:"read"`
 
@@ -630,6 +632,8 @@ type GatewayStruct struct {
 }
 
 type GatewayMethods struct {
+	ChainGetBlock func(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) ``
+
 	ChainGetBlockMessages func(p0 context.Context, p1 cid.Cid) (*BlockMessages, error) ``
 
 	ChainGetGenesis func(p0 context.Context) (*types.TipSet, error) ``
@@ -722,6 +726,8 @@ type GatewayMethods struct {
 
 	EthSubscribe func(p0 context.Context, p1 jsonrpc.RawParams) (ethtypes.EthSubscriptionID, error) ``
 
+	EthSyncing func(p0 context.Context) (ethtypes.EthSyncingResult, error) ``
+
 	EthUninstallFilter func(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) ``
 
 	EthUnsubscribe func(p0 context.Context, p1 ethtypes.EthSubscriptionID) (bool, error) ``
@@ -730,8 +736,12 @@ type GatewayMethods struct {
 
 	GasEstimateMessageGas func(p0 context.Context, p1 *types.Message, p2 *MessageSendSpec, p3 types.TipSetKey) (*types.Message, error) ``
 
+	MinerGetBaseInfo func(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) ``
+
 	MpoolGetNonce func(p0 context.Context, p1 address.Address) (uint64, error) ``
 
+	MpoolPending func(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) ``
+
 	MpoolPush func(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) ``
 
 	MsigGetAvailableBalance func(p0 context.Context, p1 address.Address, p2 types.TipSetKey) (types.BigInt, error) ``
@@ -2422,6 +2432,17 @@ func (s *FullNodeStub) EthSubscribe(p0 context.Context, p1 jsonrpc.RawParams) (e
 	return *new(ethtypes.EthSubscriptionID), ErrNotSupported
 }
 
+func (s *FullNodeStruct) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	if s.Internal.EthSyncing == nil {
+		return *new(ethtypes.EthSyncingResult), ErrNotSupported
+	}
+	return s.Internal.EthSyncing(p0)
+}
+
+func (s *FullNodeStub) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	return *new(ethtypes.EthSyncingResult), ErrNotSupported
+}
+
 func (s *FullNodeStruct) EthUninstallFilter(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) {
 	if s.Internal.EthUninstallFilter == nil {
 		return false, ErrNotSupported
@@ -4094,6 +4115,17 @@ func (s *FullNodeStub) Web3ClientVersion(p0 context.Context) (string, error) {
 	return "", ErrNotSupported
 }
 
+func (s *GatewayStruct) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	if s.Internal.ChainGetBlock == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.ChainGetBlock(p0, p1)
+}
+
+func (s *GatewayStub) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) ChainGetBlockMessages(p0 context.Context, p1 cid.Cid) (*BlockMessages, error) {
 	if s.Internal.ChainGetBlockMessages == nil {
 		return nil, ErrNotSupported
@@ -4600,6 +4632,17 @@ func (s *GatewayStub) EthSubscribe(p0 context.Context, p1 jsonrpc.RawParams) (et
 	return *new(ethtypes.EthSubscriptionID), ErrNotSupported
 }
 
+func (s *GatewayStruct) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	if s.Internal.EthSyncing == nil {
+		return *new(ethtypes.EthSyncingResult), ErrNotSupported
+	}
+	return s.Internal.EthSyncing(p0)
+}
+
+func (s *GatewayStub) EthSyncing(p0 context.Context) (ethtypes.EthSyncingResult, error) {
+	return *new(ethtypes.EthSyncingResult), ErrNotSupported
+}
+
 func (s *GatewayStruct) EthUninstallFilter(p0 context.Context, p1 ethtypes.EthFilterID) (bool, error) {
 	if s.Internal.EthUninstallFilter == nil {
 		return false, ErrNotSupported
@@ -4644,6 +4687,17 @@ func (s *GatewayStub) GasEstimateMessageGas(p0 context.Context, p1 *types.Messag
 	return nil, ErrNotSupported
 }
 
+func (s *GatewayStruct) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) {
+	if s.Internal.MinerGetBaseInfo == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.MinerGetBaseInfo(p0, p1, p2, p3)
+}
+
+func (s *GatewayStub) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*MiningBaseInfo, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolGetNonce(p0 context.Context, p1 address.Address) (uint64, error) {
 	if s.Internal.MpoolGetNonce == nil {
 		return 0, ErrNotSupported
@@ -4655,6 +4709,17 @@ func (s *GatewayStub) MpoolGetNonce(p0 context.Context, p1 address.Address) (uin
 	return 0, ErrNotSupported
 }
 
+func (s *GatewayStruct) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	if s.Internal.MpoolPending == nil {
+		return *new([]*types.SignedMessage), ErrNotSupported
+	}
+	return s.Internal.MpoolPending(p0, p1)
+}
+
+func (s *GatewayStub) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	return *new([]*types.SignedMessage), ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolPush(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) {
 	if s.Internal.MpoolPush == nil {
 		return *new(cid.Cid), ErrNotSupported

api/types.go

@@ -299,6 +299,7 @@ type MinerInfo struct {
 	SectorSize                 abi.SectorSize
 	WindowPoStPartitionSectors uint64
 	ConsensusFaultElapsed      abi.ChainEpoch
+	PendingOwnerAddress        *address.Address
 	Beneficiary                address.Address
 	BeneficiaryTerm            *miner.BeneficiaryTerm
 	PendingBeneficiaryTerm     *miner.PendingBeneficiaryChange
@@ -314,31 +315,30 @@ type NetworkParams struct {
 }
 
 type ForkUpgradeParams struct {
-	UpgradeSmokeHeight         abi.ChainEpoch
-	UpgradeBreezeHeight        abi.ChainEpoch
-	UpgradeIgnitionHeight      abi.ChainEpoch
-	UpgradeLiftoffHeight       abi.ChainEpoch
-	UpgradeAssemblyHeight      abi.ChainEpoch
-	UpgradeRefuelHeight        abi.ChainEpoch
-	UpgradeTapeHeight          abi.ChainEpoch
-	UpgradeKumquatHeight       abi.ChainEpoch
-	UpgradePriceListOopsHeight abi.ChainEpoch
-	BreezeGasTampingDuration   abi.ChainEpoch
-	UpgradeCalicoHeight        abi.ChainEpoch
-	UpgradePersianHeight       abi.ChainEpoch
-	UpgradeOrangeHeight        abi.ChainEpoch
-	UpgradeClausHeight         abi.ChainEpoch
-	UpgradeTrustHeight         abi.ChainEpoch
-	UpgradeNorwegianHeight     abi.ChainEpoch
-	UpgradeTurboHeight         abi.ChainEpoch
-	UpgradeHyperdriveHeight    abi.ChainEpoch
-	UpgradeChocolateHeight     abi.ChainEpoch
-	UpgradeOhSnapHeight        abi.ChainEpoch
-	UpgradeSkyrHeight          abi.ChainEpoch
-	UpgradeSharkHeight         abi.ChainEpoch
-	UpgradeHyggeHeight         abi.ChainEpoch
-	UpgradeLightningHeight     abi.ChainEpoch
-	UpgradeThunderHeight       abi.ChainEpoch
+	UpgradeSmokeHeight       abi.ChainEpoch
+	UpgradeBreezeHeight      abi.ChainEpoch
+	UpgradeIgnitionHeight    abi.ChainEpoch
+	UpgradeLiftoffHeight     abi.ChainEpoch
+	UpgradeAssemblyHeight    abi.ChainEpoch
+	UpgradeRefuelHeight      abi.ChainEpoch
+	UpgradeTapeHeight        abi.ChainEpoch
+	UpgradeKumquatHeight     abi.ChainEpoch
+	BreezeGasTampingDuration abi.ChainEpoch
+	UpgradeCalicoHeight      abi.ChainEpoch
+	UpgradePersianHeight     abi.ChainEpoch
+	UpgradeOrangeHeight      abi.ChainEpoch
+	UpgradeClausHeight       abi.ChainEpoch
+	UpgradeTrustHeight       abi.ChainEpoch
+	UpgradeNorwegianHeight   abi.ChainEpoch
+	UpgradeTurboHeight       abi.ChainEpoch
+	UpgradeHyperdriveHeight  abi.ChainEpoch
+	UpgradeChocolateHeight   abi.ChainEpoch
+	UpgradeOhSnapHeight      abi.ChainEpoch
+	UpgradeSkyrHeight        abi.ChainEpoch
+	UpgradeSharkHeight       abi.ChainEpoch
+	UpgradeHyggeHeight       abi.ChainEpoch
+	UpgradeLightningHeight   abi.ChainEpoch
+	UpgradeThunderHeight     abi.ChainEpoch
 }
 
 type NonceMapType map[address.Address]uint64

api/v0api/gateway.go

@@ -35,6 +35,9 @@ import (
 //  * Generate openrpc blobs
 
 type Gateway interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*api.MiningBaseInfo, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (api.MinerSectors, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*api.InvocResult, error)

api/v0api/proxy_gen.go

@@ -431,6 +431,8 @@ type GatewayStruct struct {
 }
 
 type GatewayMethods struct {
+	ChainGetBlock func(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) ``
+
 	ChainGetBlockMessages func(p0 context.Context, p1 cid.Cid) (*api.BlockMessages, error) ``
 
 	ChainGetMessage func(p0 context.Context, p1 cid.Cid) (*types.Message, error) ``
@@ -453,8 +455,12 @@ type GatewayMethods struct {
 
 	GasEstimateMessageGas func(p0 context.Context, p1 *types.Message, p2 *api.MessageSendSpec, p3 types.TipSetKey) (*types.Message, error) ``
 
+	MinerGetBaseInfo func(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) ``
+
 	MpoolGetNonce func(p0 context.Context, p1 address.Address) (uint64, error) ``
 
+	MpoolPending func(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) ``
+
 	MpoolPush func(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) ``
 
 	MsigGetAvailableBalance func(p0 context.Context, p1 address.Address, p2 types.TipSetKey) (types.BigInt, error) ``
@@ -2581,6 +2587,17 @@ func (s *FullNodeStub) WalletVerify(p0 context.Context, p1 address.Address, p2 [
 	return false, ErrNotSupported
 }
 
+func (s *GatewayStruct) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	if s.Internal.ChainGetBlock == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.ChainGetBlock(p0, p1)
+}
+
+func (s *GatewayStub) ChainGetBlock(p0 context.Context, p1 cid.Cid) (*types.BlockHeader, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) ChainGetBlockMessages(p0 context.Context, p1 cid.Cid) (*api.BlockMessages, error) {
 	if s.Internal.ChainGetBlockMessages == nil {
 		return nil, ErrNotSupported
@@ -2702,6 +2719,17 @@ func (s *GatewayStub) GasEstimateMessageGas(p0 context.Context, p1 *types.Messag
 	return nil, ErrNotSupported
 }
 
+func (s *GatewayStruct) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) {
+	if s.Internal.MinerGetBaseInfo == nil {
+		return nil, ErrNotSupported
+	}
+	return s.Internal.MinerGetBaseInfo(p0, p1, p2, p3)
+}
+
+func (s *GatewayStub) MinerGetBaseInfo(p0 context.Context, p1 address.Address, p2 abi.ChainEpoch, p3 types.TipSetKey) (*api.MiningBaseInfo, error) {
+	return nil, ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolGetNonce(p0 context.Context, p1 address.Address) (uint64, error) {
 	if s.Internal.MpoolGetNonce == nil {
 		return 0, ErrNotSupported
@@ -2713,6 +2741,17 @@ func (s *GatewayStub) MpoolGetNonce(p0 context.Context, p1 address.Address) (uin
 	return 0, ErrNotSupported
 }
 
+func (s *GatewayStruct) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	if s.Internal.MpoolPending == nil {
+		return *new([]*types.SignedMessage), ErrNotSupported
+	}
+	return s.Internal.MpoolPending(p0, p1)
+}
+
+func (s *GatewayStub) MpoolPending(p0 context.Context, p1 types.TipSetKey) ([]*types.SignedMessage, error) {
+	return *new([]*types.SignedMessage), ErrNotSupported
+}
+
 func (s *GatewayStruct) MpoolPush(p0 context.Context, p1 *types.SignedMessage) (cid.Cid, error) {
 	if s.Internal.MpoolPush == nil {
 		return *new(cid.Cid), ErrNotSupported

blockstore/badger/blockstore.go

@@ -660,6 +660,60 @@ func (b *Blockstore) Has(ctx context.Context, cid cid.Cid) (bool, error) {
 	}
 }
 
+// GetMany implements a GetMany method for the badgerDB blockstore
+// This method returns empty slices and an error if an error other than ErrKeyNotFound is encountered
+// For every ErrKeyNotFound, the missing CID is returned but the rest of the results are returned without error
+func (b *Blockstore) GetMany(ctx context.Context, cids []cid.Cid) ([]blocks.Block, []cid.Cid, error) {
+	if err := b.access(); err != nil {
+		return nil, nil, err
+	}
+	defer b.viewers.Done()
+
+	b.lockDB()
+	defer b.unlockDB()
+
+	blks := make([]blocks.Block, 0, len(cids))
+	missingCids := make([]cid.Cid, 0, len(cids))
+	err := b.db.View(func(txn *badger.Txn) error {
+		f := func(c cid.Cid) error { // encapsulate in an anonymous function to handle key pool deferral for each key
+			if !c.Defined() {
+				return ipld.ErrNotFound{Cid: c}
+			}
+			k, pooled := b.PooledStorageKey(c)
+			if pooled {
+				defer KeyPool.Put(k)
+			}
+			switch item, err := txn.Get(k); err {
+			case nil:
+				val, err := item.ValueCopy(nil)
+				if err != nil {
+					return err
+				}
+				blk, err := blocks.NewBlockWithCid(val, c)
+				if err != nil {
+					return err
+				}
+				blks = append(blks, blk)
+			case badger.ErrKeyNotFound:
+				missingCids = append(missingCids, c)
+			default:
+				return fmt.Errorf("failed to get block from badger blockstore: %w", err)
+			}
+			return nil
+		}
+		for _, c := range cids {
+			if err := f(c); err != nil {
+				return err
+			}
+		}
+		return nil
+	})
+	if err != nil {
+		return nil, nil, err
+	}
+	return blks, missingCids, nil
+}
+
 // Get implements Blockstore.Get.
 func (b *Blockstore) Get(ctx context.Context, cid cid.Cid) (blocks.Block, error) {
 	if !cid.Defined() {

blockstore/splitstore/splitstore.go

@@ -164,7 +164,7 @@ type SplitStore struct {
 	path string
 
 	mx          sync.Mutex
-	warmupEpoch abi.ChainEpoch // protected by mx
+	warmupEpoch atomic.Int64
 	baseEpoch   abi.ChainEpoch // protected by compaction lock
 	pruneEpoch  abi.ChainEpoch // protected by compaction lock
 
@@ -684,9 +684,7 @@ func (s *SplitStore) View(ctx context.Context, cid cid.Cid, cb func([]byte) erro
 }
 
 func (s *SplitStore) isWarm() bool {
-	s.mx.Lock()
-	defer s.mx.Unlock()
-	return s.warmupEpoch > 0
+	return s.warmupEpoch.Load() > 0
 }
 
 // State tracking
@@ -757,7 +755,7 @@ func (s *SplitStore) Start(chain ChainAccessor, us stmgr.UpgradeSchedule) error
 	bs, err = s.ds.Get(s.ctx, warmupEpochKey)
 	switch err {
 	case nil:
-		s.warmupEpoch = bytesToEpoch(bs)
+		s.warmupEpoch.Store(bytesToInt64(bs))
 
 	case dstore.ErrNotFound:
 		warmup = true
@@ -791,7 +789,7 @@ func (s *SplitStore) Start(chain ChainAccessor, us stmgr.UpgradeSchedule) error
 		return xerrors.Errorf("error loading compaction index: %w", err)
 	}
 
-	log.Infow("starting splitstore", "baseEpoch", s.baseEpoch, "warmupEpoch", s.warmupEpoch)
+	log.Infow("starting splitstore", "baseEpoch", s.baseEpoch, "warmupEpoch", s.warmupEpoch.Load())
 
 	if warmup {
 		err = s.warmup(curTs)

blockstore/splitstore/splitstore_check.go

@@ -145,7 +145,7 @@ func (s *SplitStore) doCheck(curTs *types.TipSet) error {
 func (s *SplitStore) Info() map[string]interface{} {
 	info := make(map[string]interface{})
 	info["base epoch"] = s.baseEpoch
-	info["warmup epoch"] = s.warmupEpoch
+	info["warmup epoch"] = s.warmupEpoch.Load()
 	info["compactions"] = s.compactionIndex
 	info["prunes"] = s.pruneIndex
 	info["compacting"] = s.compacting == 1

blockstore/splitstore/splitstore_compact.go

@@ -1114,13 +1114,17 @@ func (s *SplitStore) walkChain(ts *types.TipSet, inclState, inclMsgs abi.ChainEp
 					if err := walkBlock(c); err != nil {
 						return xerrors.Errorf("error walking block (cid: %s): %w", c, err)
 					}
+
+					if err := s.checkYield(); err != nil {
+						return xerrors.Errorf("check yield: %w", err)
+					}
 				}
 				return nil
 			})
 		}
 
 		if err := g.Wait(); err != nil {
-			return err
+			return xerrors.Errorf("walkBlock workers errored: %w", err)
 		}
 	}
 
@@ -1153,8 +1157,8 @@ func (s *SplitStore) walkObject(c cid.Cid, visitor ObjectVisitor, f func(cid.Cid
 	}
 
 	// check this before recursing
-	if err := s.checkYield(); err != nil {
-		return 0, err
+	if err := s.checkClosing(); err != nil {
+		return 0, xerrors.Errorf("check closing: %w", err)
 	}
 
 	var links []cid.Cid
@@ -1222,8 +1226,8 @@ func (s *SplitStore) walkObjectIncomplete(c cid.Cid, visitor ObjectVisitor, f, m
 	}
 
 	// check this before recursing
-	if err := s.checkYield(); err != nil {
-		return sz, err
+	if err := s.checkClosing(); err != nil {
+		return sz, xerrors.Errorf("check closing: %w", err)
 	}
 
 	var links []cid.Cid

blockstore/splitstore/splitstore_test.go

@@ -429,7 +429,7 @@ func testSplitStoreReification(t *testing.T, f func(context.Context, blockstore.
 	}
 	defer ss.Close() //nolint
 
-	ss.warmupEpoch = 1
+	ss.warmupEpoch.Store(1)
 	go ss.reifyOrchestrator()
 
 	waitForReification := func() {
@@ -529,7 +529,7 @@ func testSplitStoreReificationLimit(t *testing.T, f func(context.Context, blocks
 	}
 	defer ss.Close() //nolint
 
-	ss.warmupEpoch = 1
+	ss.warmupEpoch.Store(1)
 	go ss.reifyOrchestrator()
 
 	waitForReification := func() {

blockstore/splitstore/splitstore_warmup.go

@@ -136,9 +136,8 @@ func (s *SplitStore) doWarmup(curTs *types.TipSet) error {
 	if err != nil {
 		return xerrors.Errorf("error saving warm up epoch: %w", err)
 	}
-	s.mx.Lock()
-	s.warmupEpoch = epoch
-	s.mx.Unlock()
+
+	s.warmupEpoch.Store(int64(epoch))
 
 	// also save the compactionIndex, as this is used as an indicator of warmup for upgraded nodes
 	err = s.ds.Put(s.ctx, compactionIndexKey, int64ToBytes(s.compactionIndex))

build/params_interop.go

@@ -49,16 +49,11 @@ var UpgradeHyperdriveHeight = abi.ChainEpoch(-16)
 var UpgradeChocolateHeight = abi.ChainEpoch(-17)
 var UpgradeOhSnapHeight = abi.ChainEpoch(-18)
 var UpgradeSkyrHeight = abi.ChainEpoch(-19)
+var UpgradeSharkHeight = abi.ChainEpoch(-20)
+var UpgradeHyggeHeight = abi.ChainEpoch(-21)
+var UpgradeLightningHeight = abi.ChainEpoch(-22)
 
-const UpgradeSharkHeight = abi.ChainEpoch(-20)
-
-const UpgradeHyggeHeight = abi.ChainEpoch(100)
-
-// ??????????
-const UpgradeLightningHeight = 200
-
-// ??????????????????
-const UpgradeThunderHeight = 300
+const UpgradeThunderHeight = 50
 
 var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
 	0: DrandMainnet,

build/version.go

@@ -37,7 +37,7 @@ func BuildTypeString() string {
 }
 
 // BuildVersion is the local build version
-const BuildVersion = "1.23.2"
+const BuildVersion = "1.23.2-dev"
 
 func UserVersion() string {
 	if os.Getenv("LOTUS_VERSION_IGNORE_COMMIT") == "1" {

chain/actors/policy/policy.go

@@ -61,6 +61,10 @@ const (
 	MaxPreCommitRandomnessLookback = builtin11.EpochsInDay + SealRandomnessLookback
 )
 
+var (
+	MarketDefaultAllocationTermBuffer = market11.MarketDefaultAllocationTermBuffer
+)
+
 // SetSupportedProofTypes sets supported proof types, across all actor versions.
 // This should only be used for testing.
 func SetSupportedProofTypes(types ...abi.RegisteredSealProof) {

chain/actors/policy/policy.go.template

@@ -39,6 +39,10 @@ const (
 	MaxPreCommitRandomnessLookback = builtin{{.latestVersion}}.EpochsInDay + SealRandomnessLookback
 )
 
+var (
+	MarketDefaultAllocationTermBuffer = market{{.latestVersion}}.MarketDefaultAllocationTermBuffer
+)
+
 // SetSupportedProofTypes sets supported proof types, across all actor versions.
 // This should only be used for testing.
 func SetSupportedProofTypes(types ...abi.RegisteredSealProof) {

chain/events/observer.go

@@ -125,7 +125,7 @@ func (o *observer) listenHeadChangesOnce(ctx context.Context) error {
 
 	for changes := range notifs {
 		if err := o.applyChanges(ctx, changes); err != nil {
-			return err
+			return xerrors.Errorf("failed to apply a change notification: %w", err)
 		}
 	}
 

chain/gen/slashfilter/slashfilter.go

@@ -11,7 +11,6 @@ import (
 
 	"github.com/filecoin-project/go-state-types/abi"
 
-	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
 )
 
@@ -27,24 +26,20 @@ func New(dstore ds.Batching) *SlashFilter {
 	}
 }
 
-func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, parentEpoch abi.ChainEpoch) error {
-	if build.IsNearUpgrade(bh.Height, build.UpgradeOrangeHeight) {
-		return nil
-	}
-
+func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, parentEpoch abi.ChainEpoch) (cid.Cid, error) {
 	epochKey := ds.NewKey(fmt.Sprintf("/%s/%d", bh.Miner, bh.Height))
 	{
 		// double-fork mining (2 blocks at one epoch)
-		if err := checkFault(ctx, f.byEpoch, epochKey, bh, "double-fork mining faults"); err != nil {
-			return err
+		if witness, err := checkFault(ctx, f.byEpoch, epochKey, bh, "double-fork mining faults"); err != nil {
+			return witness, xerrors.Errorf("check double-fork mining faults: %w", err)
 		}
 	}
 
 	parentsKey := ds.NewKey(fmt.Sprintf("/%s/%x", bh.Miner, types.NewTipSetKey(bh.Parents...).Bytes()))
 	{
 		// time-offset mining faults (2 blocks with the same parents)
-		if err := checkFault(ctx, f.byParents, parentsKey, bh, "time-offset mining faults"); err != nil {
-			return err
+		if witness, err := checkFault(ctx, f.byParents, parentsKey, bh, "time-offset mining faults"); err != nil {
+			return witness, xerrors.Errorf("check time-offset mining faults: %w", err)
 		}
 	}
 
@@ -55,19 +50,19 @@ func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, par
 		parentEpochKey := ds.NewKey(fmt.Sprintf("/%s/%d", bh.Miner, parentEpoch))
 		have, err := f.byEpoch.Has(ctx, parentEpochKey)
 		if err != nil {
-			return err
+			return cid.Undef, err
 		}
 
 		if have {
 			// If we had, make sure it's in our parent tipset
 			cidb, err := f.byEpoch.Get(ctx, parentEpochKey)
 			if err != nil {
-				return xerrors.Errorf("getting other block cid: %w", err)
+				return cid.Undef, xerrors.Errorf("getting other block cid: %w", err)
 			}
 
 			_, parent, err := cid.CidFromBytes(cidb)
 			if err != nil {
-				return err
+				return cid.Undef, err
 			}
 
 			var found bool
@@ -78,45 +73,45 @@ func (f *SlashFilter) MinedBlock(ctx context.Context, bh *types.BlockHeader, par
 			}
 
 			if !found {
-				return xerrors.Errorf("produced block would trigger 'parent-grinding fault' consensus fault; miner: %s; bh: %s, expected parent: %s", bh.Miner, bh.Cid(), parent)
+				return parent, xerrors.Errorf("produced block would trigger 'parent-grinding fault' consensus fault; miner: %s; bh: %s, expected parent: %s", bh.Miner, bh.Cid(), parent)
 			}
 		}
 	}
 
 	if err := f.byParents.Put(ctx, parentsKey, bh.Cid().Bytes()); err != nil {
-		return xerrors.Errorf("putting byEpoch entry: %w", err)
+		return cid.Undef, xerrors.Errorf("putting byEpoch entry: %w", err)
 	}
 
 	if err := f.byEpoch.Put(ctx, epochKey, bh.Cid().Bytes()); err != nil {
-		return xerrors.Errorf("putting byEpoch entry: %w", err)
+		return cid.Undef, xerrors.Errorf("putting byEpoch entry: %w", err)
 	}
 
-	return nil
+	return cid.Undef, nil
 }
 
-func checkFault(ctx context.Context, t ds.Datastore, key ds.Key, bh *types.BlockHeader, faultType string) error {
+func checkFault(ctx context.Context, t ds.Datastore, key ds.Key, bh *types.BlockHeader, faultType string) (cid.Cid, error) {
 	fault, err := t.Has(ctx, key)
 	if err != nil {
-		return err
+		return cid.Undef, xerrors.Errorf("failed to read from datastore: %w", err)
 	}
 
 	if fault {
 		cidb, err := t.Get(ctx, key)
 		if err != nil {
-			return xerrors.Errorf("getting other block cid: %w", err)
+			return cid.Undef, xerrors.Errorf("getting other block cid: %w", err)
 		}
 
 		_, other, err := cid.CidFromBytes(cidb)
 		if err != nil {
-			return err
+			return cid.Undef, err
 		}
 
 		if other == bh.Cid() {
-			return nil
+			return cid.Undef, nil
 		}
 
-		return xerrors.Errorf("produced block would trigger '%s' consensus fault; miner: %s; bh: %s, other: %s", faultType, bh.Miner, bh.Cid(), other)
+		return other, xerrors.Errorf("produced block would trigger '%s' consensus fault; miner: %s; bh: %s, other: %s", faultType, bh.Miner, bh.Cid(), other)
 	}
 
-	return nil
+	return cid.Undef, nil
 }

chain/index/msgindex.go

@@ -37,7 +37,17 @@ var dbDefs = []string{
 	)`,
 	`INSERT OR IGNORE INTO _meta (version) VALUES (1)`,
 }
-var dbPragmas = []string{}
+
+var dbPragmas = []string{
+	"PRAGMA synchronous = normal",
+	"PRAGMA temp_store = memory",
+	"PRAGMA mmap_size = 30000000000",
+	"PRAGMA page_size = 32768",
+	"PRAGMA auto_vacuum = NONE",
+	"PRAGMA automatic_index = OFF",
+	"PRAGMA journal_mode = WAL",
+	"PRAGMA read_uncommitted = ON",
+}
 
 const (
 	// prepared stmts

chain/index/msgindex_test.go

@@ -39,10 +39,10 @@ func TestBasicMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	t.Log("verifying index")
 	verifyIndex(t, cs, msgIndex)
 }
@@ -51,7 +51,7 @@ func TestReorgMsgIndex(t *testing.T) {
 	// slightly more nuanced test that includes reorgs
 	// 1. Create an index with mock chain store
 	// 2. Advance/Reorg the chain for a few tipsets
-	// 3. Verify that the index contains all messages with the correct tipst/epoch
+	// 3. Verify that the index contains all messages with the correct tipset/epoch
 	cs := newMockChainStore()
 	cs.genesis()
 
@@ -67,10 +67,10 @@ func TestReorgMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	// a simple reorg
 	t.Log("doing reorg")
 	reorgme := cs.curTs
@@ -80,7 +80,8 @@ func TestReorgMsgIndex(t *testing.T) {
 	reorgmeChild := cs.makeBlk()
 	err = cs.reorg([]*types.TipSet{reorgme}, []*types.TipSet{reorgmeChild})
 	require.NoError(t, err)
-	time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
+
+	waitForCoalescerAfterLastEvent()
 
 	t.Log("verifying index")
 	verifyIndex(t, cs, msgIndex)
@@ -109,10 +110,10 @@ func TestReconcileMsgIndex(t *testing.T) {
 		t.Logf("advance to epoch %d", i+1)
 		err := cs.advance()
 		require.NoError(t, err)
-		// wait for the coalescer to notify
-		time.Sleep(CoalesceMinDelay + 10*time.Millisecond)
 	}
 
+	waitForCoalescerAfterLastEvent()
+
 	// Close it and reorg
 	err = msgIndex.Close()
 	require.NoError(t, err)
@@ -296,3 +297,11 @@ func (cs *mockChainStore) GetTipSetFromKey(ctx context.Context, tsk types.TipSet
 	}
 	return ts, nil
 }
+
+func waitForCoalescerAfterLastEvent() {
+	// It can take up to CoalesceMinDelay for the coalescer timer to fire after the last event.
+	// When the timer fires, it can wait up to CoalesceMinDelay again for more events.
+	// Therefore the total wait is 2 * CoalesceMinDelay.
+	// Then we wait another second for the listener (the index) to actually process events.
+	time.Sleep(2*CoalesceMinDelay + time.Second)
+}

chain/messagepool/check.go

@@ -35,8 +35,8 @@ func (mp *MessagePool) CheckPendingMessages(ctx context.Context, from address.Ad
 	mp.lk.RLock()
 	mset, ok, err := mp.getPendingMset(ctx, from)
 	if err != nil {
-		log.Warnf("errored while getting pending mset: %w", err)
-		return nil, err
+		mp.lk.RUnlock()
+		return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 	}
 	if ok {
 		msgs = make([]*types.Message, 0, len(mset.msgs))
@@ -71,8 +71,8 @@ func (mp *MessagePool) CheckReplaceMessages(ctx context.Context, replace []*type
 			msgMap[m.From] = mmap
 			mset, ok, err := mp.getPendingMset(ctx, m.From)
 			if err != nil {
-				log.Warnf("errored while getting pending mset: %w", err)
-				return nil, err
+				mp.lk.RUnlock()
+				return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 			}
 			if ok {
 				count += len(mset.msgs)
@@ -155,8 +155,8 @@ func (mp *MessagePool) checkMessages(ctx context.Context, msgs []*types.Message,
 			mp.lk.RLock()
 			mset, ok, err := mp.getPendingMset(ctx, m.From)
 			if err != nil {
-				log.Warnf("errored while getting pending mset: %w", err)
-				return nil, err
+				mp.lk.RUnlock()
+				return nil, xerrors.Errorf("errored while getting pending mset: %w", err)
 			}
 			if ok && !interned {
 				st = &actorState{nextNonce: mset.nextNonce, requiredFunds: mset.requiredFunds}

chain/messagepool/messagepool.go

@@ -448,12 +448,8 @@ func New(ctx context.Context, api Provider, ds dtypes.MetadataDS, us stmgr.Upgra
 	return mp, nil
 }
 
-func (mp *MessagePool) TryForEachPendingMessage(f func(cid.Cid) error) error {
-	// avoid deadlocks in splitstore compaction when something else needs to access the blockstore
-	// while holding the mpool lock
-	if !mp.lk.TryLock() {
-		return xerrors.Errorf("mpool TryForEachPendingMessage: could not acquire lock")
-	}
+func (mp *MessagePool) ForEachPendingMessage(f func(cid.Cid) error) error {
+	mp.lk.Lock()
 	defer mp.lk.Unlock()
 
 	for _, mset := range mp.pending {
@@ -749,8 +745,7 @@ func (mp *MessagePool) checkMessage(ctx context.Context, m *types.SignedMessage)
 	}
 
 	if err := mp.VerifyMsgSig(m); err != nil {
-		log.Warnf("signature verification failed: %s", err)
-		return err
+		return xerrors.Errorf("signature verification failed: %s", err)
 	}
 
 	return nil
@@ -969,13 +964,11 @@ func (mp *MessagePool) addLocked(ctx context.Context, m *types.SignedMessage, st
 	}
 
 	if _, err := mp.api.PutMessage(ctx, m); err != nil {
-		log.Warnf("mpooladd cs.PutMessage failed: %s", err)
-		return err
+		return xerrors.Errorf("mpooladd cs.PutMessage failed: %s", err)
 	}
 
 	if _, err := mp.api.PutMessage(ctx, &m.Message); err != nil {
-		log.Warnf("mpooladd cs.PutMessage failed: %s", err)
-		return err
+		return xerrors.Errorf("mpooladd cs.PutMessage failed: %s", err)
 	}
 
 	// Note: If performance becomes an issue, making this getOrCreatePendingMset will save some work

chain/messagepool/messagepool_test.go

@@ -11,9 +11,11 @@ import (
 	"github.com/ipfs/go-datastore"
 	logging "github.com/ipfs/go-log/v2"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
+	big2 "github.com/filecoin-project/go-state-types/big"
 	"github.com/filecoin-project/go-state-types/crypto"
 	"github.com/filecoin-project/go-state-types/network"
 	builtin2 "github.com/filecoin-project/specs-actors/v2/actors/builtin"
@@ -524,6 +526,36 @@ func TestPruningSimple(t *testing.T) {
 	}
 }
 
+func TestGasRewardNegative(t *testing.T) {
+	var mp MessagePool
+
+	msg := types.SignedMessage{
+		Message: types.Message{
+			GasLimit:   1000,
+			GasFeeCap:  big2.NewInt(20000),
+			GasPremium: big2.NewInt(15000),
+		},
+	}
+	baseFee := big2.NewInt(30000)
+	// Over the GasPremium, but under the BaseFee
+	gr1 := mp.getGasReward(&msg, baseFee)
+
+	msg.Message.GasFeeCap = big2.NewInt(15000)
+	// Equal to GasPremium, under the BaseFee
+	gr2 := mp.getGasReward(&msg, baseFee)
+
+	msg.Message.GasFeeCap = big2.NewInt(10000)
+	// Under both GasPremium and BaseFee
+	gr3 := mp.getGasReward(&msg, baseFee)
+
+	require.True(t, gr1.Sign() < 0)
+	require.True(t, gr2.Sign() < 0)
+	require.True(t, gr3.Sign() < 0)
+
+	require.True(t, gr1.Cmp(gr2) > 0)
+	require.True(t, gr2.Cmp(gr3) > 0)
+}
+
 func TestLoadLocal(t *testing.T) {
 	tma := newTestMpoolAPI()
 	ds := datastore.NewMapDatastore()

chain/messagepool/selection_test.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"sort"
 	"testing"
+	"time"
 
 	"github.com/ipfs/go-cid"
 	"github.com/ipfs/go-datastore"
@@ -1690,3 +1691,188 @@ readLoop:
 	}
 
 }
+
+func TestRealWorldSelectionTiming(t *testing.T) {
+	//stm: @TOKEN_WALLET_NEW_001, @TOKEN_WALLET_SIGN_001, @CHAIN_MEMPOOL_SELECT_001
+
+	// load test-messages.json.gz and rewrite the messages so that
+	// 1) we map each real actor to a test actor so that we can sign the messages
+	// 2) adjust the nonces so that they start from 0
+	file, err := os.Open("test-messages2.json.gz")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	gzr, err := gzip.NewReader(file)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dec := json.NewDecoder(gzr)
+
+	var msgs []*types.SignedMessage
+	baseNonces := make(map[address.Address]uint64)
+
+readLoop:
+	for {
+		m := new(types.SignedMessage)
+		err := dec.Decode(m)
+		switch err {
+		case nil:
+			msgs = append(msgs, m)
+			nonce, ok := baseNonces[m.Message.From]
+			if !ok || m.Message.Nonce < nonce {
+				baseNonces[m.Message.From] = m.Message.Nonce
+			}
+
+		case io.EOF:
+			break readLoop
+
+		default:
+			t.Fatal(err)
+		}
+	}
+
+	actorMap := make(map[address.Address]address.Address)
+	actorWallets := make(map[address.Address]api.Wallet)
+
+	for _, m := range msgs {
+		baseNonce := baseNonces[m.Message.From]
+
+		localActor, ok := actorMap[m.Message.From]
+		if !ok {
+			w, err := wallet.NewWallet(wallet.NewMemKeyStore())
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			a, err := w.WalletNew(context.Background(), types.KTSecp256k1)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			actorMap[m.Message.From] = a
+			actorWallets[a] = w
+			localActor = a
+		}
+
+		w, ok := actorWallets[localActor]
+		if !ok {
+			t.Fatalf("failed to lookup wallet for actor %s", localActor)
+		}
+
+		m.Message.From = localActor
+		m.Message.Nonce -= baseNonce
+
+		sig, err := w.WalletSign(context.TODO(), localActor, m.Message.Cid().Bytes(), api.MsgMeta{})
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		m.Signature = *sig
+	}
+
+	mp, tma := makeTestMpool()
+
+	block := tma.nextBlockWithHeight(uint64(build.UpgradeHyggeHeight) + 10)
+	ts := mock.TipSet(block)
+	tma.applyBlock(t, block)
+
+	for _, a := range actorMap {
+		tma.setBalance(a, 1000000)
+	}
+
+	tma.baseFee = types.NewInt(800_000_000)
+
+	sort.Slice(msgs, func(i, j int) bool {
+		return msgs[i].Message.Nonce < msgs[j].Message.Nonce
+	})
+
+	// add the messages
+	for _, m := range msgs {
+		mustAdd(t, mp, m)
+	}
+
+	// do message selection and check block packing
+	minGasLimit := int64(0.9 * float64(build.BlockGasLimit))
+
+	// greedy first
+	start := time.Now()
+	selected, err := mp.SelectMessages(context.Background(), ts, 1.0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit := int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=1.0; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// high quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .8)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.8; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// mid quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .4)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.4; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// low quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.1; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+
+	// very low quality ticket
+	start = time.Now()
+	selected, err = mp.SelectMessages(context.Background(), ts, .01)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("selected %d messages in %s", len(selected), time.Since(start))
+
+	gasLimit = int64(0)
+	for _, m := range selected {
+		gasLimit += m.Message.GasLimit
+	}
+	if gasLimit < minGasLimit {
+		t.Fatalf("failed to pack with tq=0.01; packed %d, minimum packing: %d", gasLimit, minGasLimit)
+	}
+}

chain/stmgr/execute.go

@@ -131,15 +131,17 @@ func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (c
 	tsKey := ts.Key()
 
 	// check if we have the trace for this tipset in the cache
-	sm.execTraceCacheLock.Lock()
-	if entry, ok := sm.execTraceCache.Get(tsKey); ok {
-		// we have to make a deep copy since caller can modify the invocTrace
-		// and we don't want that to change what we store in cache
-		invocTraceCopy := makeDeepCopy(entry.invocTrace)
+	if execTraceCacheSize > 0 {
+		sm.execTraceCacheLock.Lock()
+		if entry, ok := sm.execTraceCache.Get(tsKey); ok {
+			// we have to make a deep copy since caller can modify the invocTrace
+			// and we don't want that to change what we store in cache
+			invocTraceCopy := makeDeepCopy(entry.invocTrace)
+			sm.execTraceCacheLock.Unlock()
+			return entry.postStateRoot, invocTraceCopy, nil
+		}
 		sm.execTraceCacheLock.Unlock()
-		return entry.postStateRoot, invocTraceCopy, nil
 	}
-	sm.execTraceCacheLock.Unlock()
 
 	var invocTrace []*api.InvocResult
 	st, err := sm.ExecutionTraceWithMonitor(ctx, ts, &InvocationTracer{trace: &invocTrace})
@@ -147,11 +149,13 @@ func (sm *StateManager) ExecutionTrace(ctx context.Context, ts *types.TipSet) (c
 		return cid.Undef, nil, err
 	}
 
-	invocTraceCopy := makeDeepCopy(invocTrace)
+	if execTraceCacheSize > 0 {
+		invocTraceCopy := makeDeepCopy(invocTrace)
 
-	sm.execTraceCacheLock.Lock()
-	sm.execTraceCache.Add(tsKey, tipSetCacheEntry{st, invocTraceCopy})
-	sm.execTraceCacheLock.Unlock()
+		sm.execTraceCacheLock.Lock()
+		sm.execTraceCache.Add(tsKey, tipSetCacheEntry{st, invocTraceCopy})
+		sm.execTraceCacheLock.Unlock()
+	}
 
 	return st, invocTrace, nil
 }

chain/stmgr/stmgr.go

@@ -3,6 +3,8 @@ package stmgr
 import (
 	"context"
 	"fmt"
+	"os"
+	"strconv"
 	"sync"
 
 	lru "github.com/hashicorp/golang-lru/v2"
@@ -40,8 +42,7 @@ import (
 const LookbackNoLimit = api.LookbackNoLimit
 const ReceiptAmtBitwidth = 3
 
-const execTraceCacheSize = 16
-
+var execTraceCacheSize = 16
 var log = logging.Logger("statemgr")
 
 type StateManagerAPI interface {
@@ -74,6 +75,17 @@ func (m *migrationResultCache) keyForMigration(root cid.Cid) dstore.Key {
 	return dstore.NewKey(kStr)
 }
 
+func init() {
+	if s := os.Getenv("LOTUS_EXEC_TRACE_CACHE_SIZE"); s != "" {
+		letc, err := strconv.Atoi(s)
+		if err != nil {
+			log.Errorf("failed to parse 'LOTUS_EXEC_TRACE_CACHE_SIZE' env var: %s", err)
+		} else {
+			execTraceCacheSize = letc
+		}
+	}
+}
+
 func (m *migrationResultCache) Get(ctx context.Context, root cid.Cid) (cid.Cid, bool, error) {
 	k := m.keyForMigration(root)
 
@@ -200,9 +212,14 @@ func NewStateManager(cs *store.ChainStore, exec Executor, sys vm.SyscallBuilder,
 		}
 	}
 
-	execTraceCache, err := lru.NewARC[types.TipSetKey, tipSetCacheEntry](execTraceCacheSize)
-	if err != nil {
-		return nil, err
+	log.Debugf("execTraceCache size: %d", execTraceCacheSize)
+	var execTraceCache *lru.ARCCache[types.TipSetKey, tipSetCacheEntry]
+	var err error
+	if execTraceCacheSize > 0 {
+		execTraceCache, err = lru.NewARC[types.TipSetKey, tipSetCacheEntry](execTraceCacheSize)
+		if err != nil {
+			return nil, err
+		}
 	}
 
 	return &StateManager{

chain/stmgr/utils.go

@@ -72,7 +72,7 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
 
 	base, trace, err := sm.ExecutionTrace(ctx, ts)
 	if err != nil {
-		return cid.Undef, nil, err
+		return cid.Undef, nil, xerrors.Errorf("failed to compute base state: %w", err)
 	}
 
 	for i := ts.Height(); i < height; i++ {
@@ -116,6 +116,21 @@ func ComputeState(ctx context.Context, sm *StateManager, height abi.ChainEpoch,
 		if ret.ExitCode != 0 {
 			log.Infof("compute state apply message %d failed (exit: %d): %s", i, ret.ExitCode, ret.ActorErr)
 		}
+
+		ir := &api.InvocResult{
+			MsgCid:         msg.Cid(),
+			Msg:            msg,
+			MsgRct:         &ret.MessageReceipt,
+			ExecutionTrace: ret.ExecutionTrace,
+			Duration:       ret.Duration,
+		}
+		if ret.ActorErr != nil {
+			ir.Error = ret.ActorErr.Error()
+		}
+		if ret.GasCosts != nil {
+			ir.GasCost = MakeMsgGasCost(msg, ret)
+		}
+		trace = append(trace, ir)
 	}
 
 	root, err := vmi.Flush(ctx)

chain/store/store.go

@@ -425,6 +425,11 @@ func (cs *ChainStore) MaybeTakeHeavierTipSet(ctx context.Context, ts *types.TipS
 	}
 
 	defer cs.heaviestLk.Unlock()
+
+	if ts.Equals(cs.heaviest) {
+		return nil
+	}
+
 	w, err := cs.weight(ctx, cs.StateBlockstore(), ts)
 	if err != nil {
 		return err

chain/sub/incoming.go

@@ -12,7 +12,7 @@ import (
 	bserv "github.com/ipfs/go-blockservice"
 	"github.com/ipfs/go-cid"
 	logging "github.com/ipfs/go-log/v2"
-	"github.com/ipni/storetheindex/announce/message"
+	"github.com/ipni/go-libipni/announce/message"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	"github.com/libp2p/go-libp2p/core/connmgr"
 	"github.com/libp2p/go-libp2p/core/peer"
@@ -358,6 +358,8 @@ func (mv *MessageValidator) Validate(ctx context.Context, pid peer.ID, msg *pubs
 			fallthrough
 		case xerrors.Is(err, messagepool.ErrNonceGap):
 			fallthrough
+		case xerrors.Is(err, messagepool.ErrGasFeeCapTooLow):
+			fallthrough
 		case xerrors.Is(err, messagepool.ErrNonceTooLow):
 			fallthrough
 		case xerrors.Is(err, messagepool.ErrExistingNonce):

chain/sub/incoming_test.go

@@ -9,7 +9,7 @@ import (
 	"github.com/golang/mock/gomock"
 	blocks "github.com/ipfs/go-block-format"
 	"github.com/ipfs/go-cid"
-	"github.com/ipni/storetheindex/announce/message"
+	"github.com/ipni/go-libipni/announce/message"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	pb "github.com/libp2p/go-libp2p-pubsub/pb"
 	"github.com/libp2p/go-libp2p/core/peer"

chain/types/blockmsg.go

@@ -2,6 +2,7 @@ package types
 
 import (
 	"bytes"
+	"fmt"
 
 	"github.com/ipfs/go-cid"
 )
@@ -14,10 +15,13 @@ type BlockMsg struct {
 
 func DecodeBlockMsg(b []byte) (*BlockMsg, error) {
 	var bm BlockMsg
-	if err := bm.UnmarshalCBOR(bytes.NewReader(b)); err != nil {
+	data := bytes.NewReader(b)
+	if err := bm.UnmarshalCBOR(data); err != nil {
 		return nil, err
 	}
-
+	if l := data.Len(); l != 0 {
+		return nil, fmt.Errorf("extraneous data in BlockMsg CBOR encoding: got %d unexpected bytes", l)
+	}
 	return &bm, nil
 }
 

chain/types/blockmsg_test.go

@@ -0,0 +1,40 @@
+package types
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestDecodeBlockMsg(t *testing.T) {
+	type args struct {
+		b []byte
+	}
+	tests := []struct {
+		name    string
+		data    []byte
+		want    *BlockMsg
+		wantErr bool
+	}{
+		{"decode empty BlockMsg with extra data at the end", []byte{0x83, 0xf6, 0x80, 0x80, 0x20}, nil, true},
+		{"decode valid empty BlockMsg", []byte{0x83, 0xf6, 0x80, 0x80}, new(BlockMsg), false},
+		{"decode invalid cbor", []byte{0x83, 0xf6, 0x80}, nil, true},
+	}
+	for _, tt := range tests {
+		data := tt.data
+		want := tt.want
+		wantErr := tt.wantErr
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := DecodeBlockMsg(data)
+			if wantErr {
+				assert.Errorf(t, err, "DecodeBlockMsg(%x)", data)
+				return
+			}
+			assert.NoErrorf(t, err, "DecodeBlockMsg(%x)", data)
+			assert.Equalf(t, want, got, "DecodeBlockMsg(%x)", data)
+			serialized, err := got.Serialize()
+			assert.NoErrorf(t, err, "DecodeBlockMsg(%x)", data)
+			assert.Equalf(t, serialized, data, "DecodeBlockMsg(%x)", data)
+		})
+	}
+}

chain/types/ethtypes/eth_types.go

@@ -238,6 +238,30 @@ func (c *EthCall) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+type EthSyncingResult struct {
+	DoneSync      bool
+	StartingBlock EthUint64
+	CurrentBlock  EthUint64
+	HighestBlock  EthUint64
+}
+
+func (sr EthSyncingResult) MarshalJSON() ([]byte, error) {
+	if sr.DoneSync {
+		// when done syncing, the json response should be '"result": false'
+		return []byte("false"), nil
+	}
+
+	// need to do an anonymous struct to avoid infinite recursion
+	return json.Marshal(&struct {
+		StartingBlock EthUint64 `json:"startingblock"`
+		CurrentBlock  EthUint64 `json:"currentblock"`
+		HighestBlock  EthUint64 `json:"highestblock"`
+	}{
+		StartingBlock: sr.StartingBlock,
+		CurrentBlock:  sr.CurrentBlock,
+		HighestBlock:  sr.HighestBlock})
+}
+
 const (
 	EthAddressLength = 20
 	EthHashLength    = 32
@@ -548,12 +572,12 @@ func (h EthSubscriptionID) String() string {
 }
 
 type EthFilterSpec struct {
-	// Interpreted as an epoch or one of "latest" for last mined block, "earliest" for first,
+	// Interpreted as an epoch (in hex) or one of "latest" for last mined block, "earliest" for first,
 	// "pending" for not yet committed messages.
 	// Optional, default: "latest".
 	FromBlock *string `json:"fromBlock,omitempty"`
 
-	// Interpreted as an epoch or one of "latest" for last mined block, "earliest" for first,
+	// Interpreted as an epoch (in hex) or one of "latest" for last mined block, "earliest" for first,
 	// "pending" for not yet committed messages.
 	// Optional, default: "latest".
 	ToBlock *string `json:"toBlock,omitempty"`

cli/info.go

@@ -20,6 +20,7 @@ import (
 	"github.com/filecoin-project/lotus/api/v1api"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/journal/alerting"
 )
 
 var infoCmd = &cli.Command{
@@ -62,6 +63,21 @@ func infoCmdAct(cctx *cli.Context) error {
 	fmt.Printf(" [epoch %s]\n", color.MagentaString(("%d"), status.SyncStatus.Epoch))
 	fmt.Printf("Peers to: [publish messages %d] [publish blocks %d]\n", status.PeerStatus.PeersToPublishMsgs, status.PeerStatus.PeersToPublishBlocks)
 
+	alerts, err := fullapi.LogAlerts(ctx)
+	if err != nil {
+		fmt.Printf("ERROR: getting alerts: %s\n", err)
+	}
+
+	activeAlerts := make([]alerting.Alert, 0)
+	for _, alert := range alerts {
+		if alert.Active {
+			activeAlerts = append(activeAlerts, alert)
+		}
+	}
+	if len(activeAlerts) > 0 {
+		fmt.Printf("%s (check %s)\n", color.RedString("⚠ %d Active alerts", len(activeAlerts)), color.YellowString("lotus log alerts"))
+	}
+
 	//Chain health calculated as percentage: amount of blocks in last finality / very healthy amount of blocks in a finality (900 epochs * 5 blocks per tipset)
 	health := (100 * (900 * status.ChainStatus.BlocksPerTipsetLastFinality) / (900 * 5))
 	switch {

cli/state.go

@@ -1065,12 +1065,19 @@ var StateComputeStateCmd = &cli.Command{
 
 		ctx := ReqContext(cctx)
 
-		ts, err := LoadTipSet(ctx, cctx, api)
+		h := abi.ChainEpoch(cctx.Uint64("vm-height"))
+		var ts *types.TipSet
+		if tss := cctx.String("tipset"); tss != "" {
+			ts, err = ParseTipSetRef(ctx, api, tss)
+		} else if h > 0 {
+			ts, err = api.ChainGetTipSetByHeight(ctx, h, types.EmptyTSK)
+		} else {
+			ts, err = api.ChainHead(ctx)
+		}
 		if err != nil {
 			return err
 		}
 
-		h := abi.ChainEpoch(cctx.Uint64("vm-height"))
 		if h == 0 {
 			h = ts.Height()
 		}
@@ -1528,6 +1535,9 @@ func printMsg(ctx context.Context, api v0api.FullNode, msg cid.Cid, mw *lapi.Msg
 	if err := printReceiptReturn(ctx, api, m, mw.Receipt); err != nil {
 		return err
 	}
+	if mw.Receipt.EventsRoot != nil {
+		fmt.Printf("Events Root: %s\n", mw.Receipt.EventsRoot)
+	}
 
 	return nil
 }

cmd/lotus-bench/main.go

@@ -98,14 +98,16 @@ func main() {
 	log.Info("Starting lotus-bench")
 
 	app := &cli.App{
-		Name:    "lotus-bench",
-		Usage:   "Benchmark performance of lotus on your hardware",
-		Version: build.UserVersion(),
+		Name:                      "lotus-bench",
+		Usage:                     "Benchmark performance of lotus on your hardware",
+		Version:                   build.UserVersion(),
+		DisableSliceFlagSeparator: true,
 		Commands: []*cli.Command{
 			proveCmd,
 			sealBenchCmd,
 			simpleCmd,
 			importBenchCmd,
+			rpcCmd,
 		},
 	}
 

cmd/lotus-bench/rpc.go

@@ -0,0 +1,576 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"os/signal"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"text/tabwriter"
+	"time"
+
+	"github.com/urfave/cli/v2"
+)
+
+var rpcCmd = &cli.Command{
+	Name:  "rpc",
+	Usage: "Runs a concurrent stress test on one or more rpc methods and prints the performance metrics including latency distribution and histogram",
+	Description: `This benchmark is designed to stress test the rpc methods of a lotus node so that we can simulate real world usage and measure the performance of rpc methods on the node.
+	
+This benchmark has the following features:
+* Can query each method both sequentially and concurrently
+* Supports rate limiting
+* Can query multiple different endpoints at once (supporting different concurrency level and rate limiting for each method)
+* Gives a nice reporting summary of the stress testing of each method (including latency distribution, histogram and more)
+* Easy to use
+
+To use this benchmark you must specify the rpc methods you want to test using the --method options, the format of it is:
+
+  --method=NAME[:CONCURRENCY][:QPS][:PARAMS] where only NAME is required.
+
+Here are some real examples:
+  lotus-bench rpc --method='eth_chainId' // run eth_chainId with default concurrency and qps
+  lotus-bench rpc --method='eth_chainId:3'  // override concurrency to 3
+  lotus-bench rpc --method='eth_chainId::100' // override to 100 qps while using default concurrency
+  lotus-bench rpc --method='eth_chainId:3:100' // run using 3 workers but limit to 100 qps
+  lotus-bench rpc --method='eth_getTransactionCount:::["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run using optional params while using default concurrency and qps
+  lotus-bench rpc --method='eth_chainId' --method='eth_getTransactionCount:10:0:["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run multiple methods at once
+
+NOTE: The last two examples will not work until we upgrade urfave dependency (tracked in https://github.com/urfave/cli/issues/1734)`,
+	Flags: []cli.Flag{
+		&cli.StringFlag{
+			Name:  "endpoint",
+			Value: "http://127.0.0.1:1234/rpc/v1",
+			Usage: "The rpc endpoint to benchmark",
+		},
+		&cli.DurationFlag{
+			Name:  "duration",
+			Value: 60 * time.Second,
+			Usage: "Duration of benchmark in seconds",
+		},
+		&cli.IntFlag{
+			Name:  "concurrency",
+			Value: 10,
+			Usage: "How many workers should be used per rpc method (can be overridden per method)",
+		},
+		&cli.IntFlag{
+			Name:  "qps",
+			Value: 0,
+			Usage: "How many requests per second should be sent per rpc method (can be overridden per method), a value of 0 means no limit",
+		},
+		&cli.StringSliceFlag{
+			Name: "method",
+			Usage: `Method to benchmark, you can specify multiple methods by repeating this flag. You can also specify method specific options to set the concurrency and qps for each method (see usage).
+`,
+		},
+		&cli.DurationFlag{
+			Name:  "watch",
+			Value: 0 * time.Second,
+			Usage: "If >0 then generates reports every N seconds (only supports linux/unix)",
+		},
+		&cli.BoolFlag{
+			Name:  "print-response",
+			Value: false,
+			Usage: "print the response of each request",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		if len(cctx.StringSlice("method")) == 0 {
+			return errors.New("you must specify and least one method to benchmark")
+		}
+
+		var rpcMethods []*RPCMethod
+		for _, str := range cctx.StringSlice("method") {
+			entries := strings.SplitN(str, ":", 4)
+			if len(entries) == 0 {
+				return errors.New("invalid method format")
+			}
+
+			// check if concurrency was specified
+			concurrency := cctx.Int("concurrency")
+			if len(entries) > 1 {
+				if len(entries[1]) > 0 {
+					var err error
+					concurrency, err = strconv.Atoi(entries[1])
+					if err != nil {
+						return fmt.Errorf("could not parse concurrency value from method %s: %v", entries[0], err)
+					}
+				}
+			}
+
+			// check if qps was specified
+			qps := cctx.Int("qps")
+			if len(entries) > 2 {
+				if len(entries[2]) > 0 {
+					var err error
+					qps, err = strconv.Atoi(entries[2])
+					if err != nil {
+						return fmt.Errorf("could not parse qps value from method %s: %v", entries[0], err)
+					}
+				}
+			}
+
+			// check if params was specified
+			params := "[]"
+			if len(entries) > 3 {
+				params = entries[3]
+			}
+
+			rpcMethods = append(rpcMethods, &RPCMethod{
+				w:           os.Stdout,
+				uri:         cctx.String("endpoint"),
+				method:      entries[0],
+				concurrency: concurrency,
+				qps:         qps,
+				params:      params,
+				printResp:   cctx.Bool("print-response"),
+			})
+		}
+
+		// terminate early on ctrl+c
+		c := make(chan os.Signal, 1)
+		signal.Notify(c, os.Interrupt)
+		go func() {
+			<-c
+			fmt.Println("Received interrupt, stopping...")
+			for _, method := range rpcMethods {
+				method.Stop()
+			}
+		}()
+
+		// stop all threads after duration
+		go func() {
+			time.Sleep(cctx.Duration("duration"))
+			for _, e := range rpcMethods {
+				e.Stop()
+			}
+		}()
+
+		// start all threads
+		var wg sync.WaitGroup
+		wg.Add(len(rpcMethods))
+
+		for _, e := range rpcMethods {
+			go func(e *RPCMethod) {
+				defer wg.Done()
+				err := e.Run()
+				if err != nil {
+					fmt.Printf("error running rpc method: %v\n", err)
+				}
+			}(e)
+		}
+
+		// if watch is set then print a report every N seconds
+		var progressCh chan struct{}
+		if cctx.Duration("watch") > 0 {
+			progressCh = make(chan struct{}, 1)
+			go func(progressCh chan struct{}) {
+				ticker := time.NewTicker(cctx.Duration("watch"))
+				for {
+					clearAndPrintReport := func() {
+						// clear the screen move the curser to the top left
+						fmt.Print("\033[2J")
+						fmt.Printf("\033[%d;%dH", 1, 1)
+						for i, e := range rpcMethods {
+							e.Report()
+							if i < len(rpcMethods)-1 {
+								fmt.Println()
+							}
+						}
+					}
+					select {
+					case <-ticker.C:
+						clearAndPrintReport()
+					case <-progressCh:
+						clearAndPrintReport()
+						return
+					}
+				}
+			}(progressCh)
+		}
+
+		wg.Wait()
+
+		if progressCh != nil {
+			// wait for the watch go routine to return
+			progressCh <- struct{}{}
+
+			// no need to print the report again
+			return nil
+		}
+
+		// print the report for each endpoint
+		for i, e := range rpcMethods {
+			e.Report()
+			if i < len(rpcMethods)-1 {
+				fmt.Println()
+			}
+		}
+
+		return nil
+	},
+}
+
+// RPCMethod handles the benchmarking of a single endpoint method.
+type RPCMethod struct {
+	w io.Writer
+	// the endpoint uri
+	uri string
+	// the rpc method we want to benchmark
+	method string
+	// the number of concurrent requests to make to this endpoint
+	concurrency int
+	// if >0 then limit to qps is the max number of requests per second to make to this endpoint (0 = no limit)
+	qps int
+	// many endpoints require specific parameters to be passed
+	params string
+	// whether or not to print the response of each request (useful for debugging)
+	printResp bool
+	// instruct the worker go routines to stop
+	stopCh chan struct{}
+	// when the endpoint bencharking started
+	start time.Time
+	// results channel is used by the workers to send results to the reporter
+	results chan *result
+	// reporter handles reading the results from workers and printing the report statistics
+	reporter *Reporter
+}
+
+// result is the result of a single rpc method request.
+type result struct {
+	err        error
+	statusCode *int
+	duration   time.Duration
+}
+
+func (rpc *RPCMethod) Run() error {
+	client := &http.Client{
+		Timeout: 0,
+	}
+
+	var wg sync.WaitGroup
+	wg.Add(rpc.concurrency)
+
+	rpc.results = make(chan *result, rpc.concurrency*1_000)
+	rpc.stopCh = make(chan struct{}, rpc.concurrency)
+
+	go func() {
+		rpc.reporter = NewReporter(rpc.results, rpc.w)
+		rpc.reporter.Run()
+	}()
+
+	rpc.start = time.Now()
+
+	// throttle the number of requests per second
+	var qpsTicker *time.Ticker
+	if rpc.qps > 0 {
+		qpsTicker = time.NewTicker(time.Second / time.Duration(rpc.qps))
+	}
+
+	for i := 0; i < rpc.concurrency; i++ {
+		go func() {
+			rpc.startWorker(client, qpsTicker)
+			wg.Done()
+		}()
+	}
+	wg.Wait()
+
+	// close the results channel so reporter will stop
+	close(rpc.results)
+
+	// wait until the reporter is done
+	<-rpc.reporter.doneCh
+
+	return nil
+}
+
+func (rpc *RPCMethod) startWorker(client *http.Client, qpsTicker *time.Ticker) {
+	for {
+		// check if we should stop
+		select {
+		case <-rpc.stopCh:
+			return
+		default:
+		}
+
+		// wait for the next tick if we are rate limiting this endpoint
+		if qpsTicker != nil {
+			<-qpsTicker.C
+		}
+
+		req, err := rpc.buildRequest()
+		if err != nil {
+			log.Fatalln(err)
+		}
+
+		start := time.Now()
+
+		var statusCode *int
+
+		// send request the endpoint
+		resp, err := client.Do(req)
+		if err != nil {
+			err = fmt.Errorf("HTTP error: %s", err.Error())
+		} else {
+			statusCode = &resp.StatusCode
+
+			// there was not a HTTP error but we need to still check the json response for errrors
+			var data []byte
+			data, err = io.ReadAll(resp.Body)
+			if err != nil {
+				log.Fatalln(err)
+			}
+
+			// we are only interested if it has the error field in the response
+			type respData struct {
+				Error struct {
+					Code    int    `json:"code"`
+					Message string `json:"message"`
+				} `json:"error"`
+			}
+
+			// unmarshal the response into a struct so we can check for errors
+			var d respData
+			err = json.Unmarshal(data, &d)
+			if err != nil {
+				log.Fatalln(err)
+			}
+
+			// if the response has an error json message then it should be considered an error just like any http error
+			if len(d.Error.Message) > 0 {
+				// truncate the error message if it is too long
+				if len(d.Error.Message) > 1000 {
+					d.Error.Message = d.Error.Message[:1000] + "..."
+				}
+				// remove newlines from the error message so we don't screw up the report
+				d.Error.Message = strings.ReplaceAll(d.Error.Message, "\n", "")
+
+				err = fmt.Errorf("JSON error: code:%d, message:%s", d.Error.Code, d.Error.Message)
+			}
+
+			if rpc.printResp {
+				fmt.Printf("[%s] %s", rpc.method, string(data))
+			}
+
+			resp.Body.Close() //nolint:errcheck
+		}
+
+		rpc.results <- &result{
+			statusCode: statusCode,
+			err:        err,
+			duration:   time.Since(start),
+		}
+	}
+}
+
+func (rpc *RPCMethod) buildRequest() (*http.Request, error) {
+	jreq, err := json.Marshal(struct {
+		Jsonrpc string          `json:"jsonrpc"`
+		ID      int             `json:"id"`
+		Method  string          `json:"method"`
+		Params  json.RawMessage `json:"params"`
+	}{
+		Jsonrpc: "2.0",
+		Method:  rpc.method,
+		Params:  json.RawMessage(rpc.params),
+		ID:      0,
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	req, err := http.NewRequest("POST", rpc.uri, bytes.NewReader(jreq))
+	if err != nil {
+		return nil, err
+	}
+
+	req.Header.Set("Accept", "application/json")
+
+	return req, nil
+}
+
+func (rpc *RPCMethod) Stop() {
+	for i := 0; i < rpc.concurrency; i++ {
+		rpc.stopCh <- struct{}{}
+	}
+}
+
+func (rpc *RPCMethod) Report() {
+	total := time.Since(rpc.start)
+	fmt.Fprintf(rpc.w, "[%s]:\n", rpc.method)
+	fmt.Fprintf(rpc.w, "- Options:\n")
+	fmt.Fprintf(rpc.w, "  - concurrency: %d\n", rpc.concurrency)
+	fmt.Fprintf(rpc.w, "  - params: %s\n", rpc.params)
+	fmt.Fprintf(rpc.w, "  - qps: %d\n", rpc.qps)
+	rpc.reporter.Print(total, rpc.w)
+}
+
+// Reporter reads the results from the workers through the results channel and aggregates the results.
+type Reporter struct {
+	// write the report to this writer
+	w io.Writer
+	// the reporter read the results from this channel
+	results chan *result
+	// doneCh is used to signal that the reporter has finished reading the results (channel has closed)
+	doneCh chan bool
+
+	// lock protect the following fields during critical sections (if --watch was specified)
+	lock sync.Mutex
+	// the latencies of all requests
+	latencies []int64
+	// the number of requests that returned each status code
+	statusCodes map[int]int
+	// the number of errors that occurred
+	errors map[string]int
+}
+
+func NewReporter(results chan *result, w io.Writer) *Reporter {
+	return &Reporter{
+		w:           w,
+		results:     results,
+		doneCh:      make(chan bool, 1),
+		statusCodes: make(map[int]int),
+		errors:      make(map[string]int),
+	}
+}
+
+func (r *Reporter) Run() {
+	for res := range r.results {
+		r.lock.Lock()
+
+		r.latencies = append(r.latencies, res.duration.Milliseconds())
+
+		if res.statusCode != nil {
+			r.statusCodes[*res.statusCode]++
+		}
+
+		if res.err != nil {
+			if len(r.errors) < 1_000_000 {
+				r.errors[res.err.Error()]++
+			} else {
+				// we don't want to store too many errors in memory
+				r.errors["hidden"]++
+			}
+		} else {
+			r.errors["nil"]++
+		}
+
+		r.lock.Unlock()
+	}
+
+	r.doneCh <- true
+}
+
+func (r *Reporter) Print(elapsed time.Duration, w io.Writer) {
+	r.lock.Lock()
+	defer r.lock.Unlock()
+
+	nrReq := int64(len(r.latencies))
+	if nrReq == 0 {
+		fmt.Println("No requests were made")
+		return
+	}
+
+	// we need to sort the latencies slice to calculate the percentiles
+	sort.Slice(r.latencies, func(i, j int) bool {
+		return r.latencies[i] < r.latencies[j]
+	})
+
+	var totalLatency int64 = 0
+	for _, latency := range r.latencies {
+		totalLatency += latency
+	}
+
+	fmt.Fprintf(w, "- Total Requests: %d\n", nrReq)
+	fmt.Fprintf(w, "- Total Duration: %dms\n", elapsed.Milliseconds())
+	fmt.Fprintf(w, "- Requests/sec: %f\n", float64(nrReq)/elapsed.Seconds())
+	fmt.Fprintf(w, "- Avg latency: %dms\n", totalLatency/nrReq)
+	fmt.Fprintf(w, "- Median latency: %dms\n", r.latencies[nrReq/2])
+	fmt.Fprintf(w, "- Latency distribution:\n")
+	percentiles := []float64{0.1, 0.5, 0.9, 0.95, 0.99, 0.999}
+	for _, p := range percentiles {
+		idx := int64(p * float64(nrReq))
+		fmt.Fprintf(w, "    %s%% in %dms\n", fmt.Sprintf("%.2f", p*100.0), r.latencies[idx])
+	}
+
+	// create a simple histogram with 10 buckets spanning the range of latency
+	// into equal ranges
+	//
+	nrBucket := 10
+	buckets := make([]Bucket, nrBucket)
+	latencyRange := r.latencies[len(r.latencies)-1]
+	bucketRange := latencyRange / int64(nrBucket)
+
+	// mark the end of each bucket
+	for i := 0; i < nrBucket; i++ {
+		buckets[i].start = int64(i) * bucketRange
+		buckets[i].end = buckets[i].start + bucketRange
+		// extend the last bucked by any remaning range caused by the integer division
+		if i == nrBucket-1 {
+			buckets[i].end = latencyRange
+		}
+	}
+
+	// count the number of requests in each bucket
+	currBucket := 0
+	for i := 0; i < len(r.latencies); {
+		if r.latencies[i] <= buckets[currBucket].end {
+			buckets[currBucket].cnt++
+			i++
+		} else {
+			currBucket++
+		}
+	}
+
+	// print the histogram using a tabwriter which will align the columns nicely
+	fmt.Fprintf(w, "- Histogram:\n")
+	const padding = 2
+	tabWriter := tabwriter.NewWriter(w, 0, 0, padding, ' ', tabwriter.AlignRight|tabwriter.Debug)
+	for i := 0; i < nrBucket; i++ {
+		ratio := float64(buckets[i].cnt) / float64(nrReq)
+		bars := strings.Repeat("#", int(ratio*100))
+		fmt.Fprintf(tabWriter, "  %d-%dms\t%d\t%s (%s%%)\n", buckets[i].start, buckets[i].end, buckets[i].cnt, bars, fmt.Sprintf("%.2f", ratio*100))
+	}
+	tabWriter.Flush() //nolint:errcheck
+
+	fmt.Fprintf(w, "- Status codes:\n")
+	for code, cnt := range r.statusCodes {
+		fmt.Fprintf(w, "    [%d]: %d\n", code, cnt)
+	}
+
+	// print the 10 most occurring errors (in case error values are not unique)
+	//
+	type kv struct {
+		err string
+		cnt int
+	}
+	var sortedErrors []kv
+	for err, cnt := range r.errors {
+		sortedErrors = append(sortedErrors, kv{err, cnt})
+	}
+	sort.Slice(sortedErrors, func(i, j int) bool {
+		return sortedErrors[i].cnt > sortedErrors[j].cnt
+	})
+	fmt.Fprintf(w, "- Errors (top 10):\n")
+	for i, se := range sortedErrors {
+		if i > 10 {
+			break
+		}
+		fmt.Fprintf(w, "    [%s]: %d\n", se.err, se.cnt)
+	}
+}
+
+type Bucket struct {
+	start int64
+	// the end value of the bucket
+	end int64
+	// how many entries are in the bucket
+	cnt int
+}

cmd/lotus-fountain/main.go

@@ -7,6 +7,7 @@ import (
 	"net"
 	"net/http"
 	"os"
+	"strings"
 	"time"
 
 	rice "github.com/GeertJohan/go.rice"
@@ -15,10 +16,14 @@ import (
 	"golang.org/x/xerrors"
 
 	"github.com/filecoin-project/go-address"
+	verifregtypes9 "github.com/filecoin-project/go-state-types/builtin/v9/verifreg"
 
 	"github.com/filecoin-project/lotus/api/v0api"
 	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/chain/actors"
+	"github.com/filecoin-project/lotus/chain/actors/builtin/verifreg"
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
 	lcli "github.com/filecoin-project/lotus/cli"
 )
 
@@ -70,6 +75,11 @@ var runCmd = &cli.Command{
 			EnvVars: []string{"LOTUS_FOUNTAIN_AMOUNT"},
 			Value:   "50",
 		},
+		&cli.Uint64Flag{
+			Name:    "data-cap",
+			EnvVars: []string{"LOTUS_DATACAP_AMOUNT"},
+			Value:   verifregtypes9.MinVerifiedDealSize.Uint64(),
+		},
 		&cli.Float64Flag{
 			Name:  "captcha-threshold",
 			Value: 0.5,
@@ -108,6 +118,7 @@ var runCmd = &cli.Command{
 			ctx:            ctx,
 			api:            nodeApi,
 			from:           from,
+			allowance:      types.NewInt(cctx.Uint64("data-cap")),
 			sendPerRequest: sendPerRequest,
 			limiter: NewLimiter(LimiterConfig{
 				TotalRate:   500 * time.Millisecond,
@@ -124,6 +135,8 @@ var runCmd = &cli.Command{
 		http.Handle("/", http.FileServer(box.HTTPBox()))
 		http.HandleFunc("/funds.html", prepFundsHtml(box))
 		http.Handle("/send", h)
+		http.HandleFunc("/datacap.html", prepDataCapHtml(box))
+		http.Handle("/datacap", h)
 		fmt.Printf("Open http://%s\n", cctx.String("front"))
 
 		go func() {
@@ -156,12 +169,24 @@ func prepFundsHtml(box *rice.Box) http.HandlerFunc {
 	}
 }
 
+func prepDataCapHtml(box *rice.Box) http.HandlerFunc {
+	tmpl := template.Must(template.New("datacaps").Parse(box.MustString("datacap.html")))
+	return func(w http.ResponseWriter, r *http.Request) {
+		err := tmpl.Execute(w, os.Getenv("RECAPTCHA_SITE_KEY"))
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadGateway)
+			return
+		}
+	}
+}
+
 type handler struct {
 	ctx context.Context
 	api v0api.FullNode
 
 	from           address.Address
 	sendPerRequest types.FIL
+	allowance      types.BigInt
 
 	limiter        *Limiter
 	recapThreshold float64
@@ -187,24 +212,41 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		http.Error(w, err.Error(), http.StatusBadGateway)
 		return
 	}
+
 	if !capResp.Success || capResp.Score < h.recapThreshold {
 		log.Infow("spam", "capResp", capResp)
 		http.Error(w, "spam protection", http.StatusUnprocessableEntity)
 		return
 	}
 
-	to, err := address.NewFromString(r.FormValue("address"))
-	if err != nil {
+	addressInput := r.FormValue("address")
+
+	var filecoinAddress address.Address
+	var decodeError error
+
+	if strings.HasPrefix(addressInput, "0x") {
+		ethAddress, err := ethtypes.ParseEthAddress(addressInput)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+
+		filecoinAddress, decodeError = ethAddress.ToFilecoinAddress()
+	} else {
+		filecoinAddress, decodeError = address.NewFromString(addressInput)
+	}
+
+	if decodeError != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return
 	}
-	if to == address.Undef {
+	if filecoinAddress == address.Undef {
 		http.Error(w, "empty address", http.StatusBadRequest)
 		return
 	}
 
 	// Limit based on wallet address
-	limiter := h.limiter.GetWalletLimiter(to.String())
+	limiter := h.limiter.GetWalletLimiter(filecoinAddress.String())
 	if !limiter.Allow() {
 		http.Error(w, http.StatusText(http.StatusTooManyRequests)+": wallet limit", http.StatusTooManyRequests)
 		return
@@ -227,11 +269,37 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	smsg, err := h.api.MpoolPushMessage(h.ctx, &types.Message{
-		Value: types.BigInt(h.sendPerRequest),
-		From:  h.from,
-		To:    to,
-	}, nil)
+	var smsg *types.SignedMessage
+	if r.RequestURI == "/send" {
+		smsg, err = h.api.MpoolPushMessage(
+			h.ctx, &types.Message{
+				Value: types.BigInt(h.sendPerRequest),
+				From:  h.from,
+				To:    filecoinAddress,
+			}, nil)
+	} else if r.RequestURI == "/datacap" {
+		var params []byte
+		params, err = actors.SerializeParams(
+			&verifregtypes9.AddVerifiedClientParams{
+				Address:   filecoinAddress,
+				Allowance: h.allowance,
+			})
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		smsg, err = h.api.MpoolPushMessage(
+			h.ctx, &types.Message{
+				Params: params,
+				From:   h.from,
+				To:     verifreg.Address,
+				Method: verifreg.Methods.AddVerifiedClient,
+			}, nil)
+	} else {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
 	if err != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return

cmd/lotus-fountain/site/datacap.html

@@ -0,0 +1,41 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Grant DataCap - Lotus Fountain</title>
+    <link rel="stylesheet" type="text/css" href="main.css">
+	<script src="https://www.google.com/recaptcha/api.js"></script>
+	<script>
+   		function onSubmit(token) {
+     	document.getElementById("datacap-form").submit();
+   	}
+	</script>
+
+</head>
+<body>
+<div class="Index">
+    <div class="Index-nodes">
+        <div class="Index-node">
+            <h3>Grant datacap</h3>
+            <p>Please input your address to receive a data cap on the Calibration Testnet.</p>
+        </div>
+        <div class="Index-node">
+            <form action='/datacap' method='post' id='datacap-form'>
+                <span>Enter destination address:</span>
+                <input type='text' name='address' style="width: 300px" placeholder="t0/1/2/3/4 or 0xETH">
+                <button class="g-recaptcha"
+                        data-sitekey="{{ . }}"
+                        data-callback='onSubmit'
+                        data-action='submit'>Grant Datacap</button>
+            </form>
+        </div>
+    </div>
+
+    <div class="Index-footer">
+        <div>
+            <a href="index.html">[Back]</a>
+            <span style="float: right">Not dispensing real Filecoin tokens</span>
+        </div>
+    </div>
+</div>
+</body>
+</html>

cmd/lotus-fountain/site/funds.html

@@ -15,12 +15,13 @@
 <div class="Index">
     <div class="Index-nodes">
         <div class="Index-node">
-            [SENDING FUNDS]
+            <h3>Send funds</h3>
+            <p>Please input your address to receive test FIL (tFIL) on the Calibration Testnet. This faucet dispenses 100 tFIL.</p>
         </div>
         <div class="Index-node">
             <form action='/send' method='post' id='funds-form'>
                 <span>Enter destination address:</span>
-				<input type='text' name='address' style="width: 300px">
+                <input type='text' name='address' style="width: 300px" placeholder="Enter t0/1/2/3/4 or 0xETH">
 				<button class="g-recaptcha" 
 						data-sitekey="{{ . }}"
 						data-callback='onSubmit' 

cmd/lotus-fountain/site/index.html

@@ -8,10 +8,16 @@
 <div class="Index">
     <div class="Index-nodes">
         <div class="Index-node">
-            [LOTUS DEVNET FAUCET]
+            LOTUS DEVNET FAUCET
         </div>
         <div class="Index-node">
-            <a href="funds.html">[Send Funds]</a>
+            <a href="funds.html">Send Funds</a>
+        </div>
+        <div class="Index-node">
+            LOTUS DEVNET GRANT DATACAP
+        </div>
+        <div class="Index-node">
+            <a href="datacap.html">Grant DataCap</a>
         </div>
     </div>
     <div class="Index-footer">

cmd/lotus-fountain/site/main.css

@@ -1,7 +1,6 @@
 body {
-    font-family: 'monospace';
-    background: #1f1f1f;
-    color: #f0f0f0;
+    font-family: 'Helvetica Neue', sans-serif;
+    background-color: #f0f0f0;
     padding: 0;
     margin: 0;
 }
@@ -9,21 +8,22 @@ body {
 .Index {
     width: 100vw;
     height: 100vh;
-    background: #1a1a1a;
-    color: #f0f0f0;
-    font-family: monospace;
+    background-color: #f0f0f0;
+    color: #333;
+    font-family: 'Helvetica Neue', sans-serif;
 
     display: grid;
     grid-template-columns: auto 40vw auto;
     grid-template-rows: auto auto auto 3em;
     grid-template-areas:
-            ". . . ."
-            ". main main ."
-            ". . . ."
-            "footer footer footer footer";
+    ". . . ."
+    ". main main ."
+    ". . . ."
+    "footer footer footer footer";
 }
+
 .Index-footer {
-    background: #2a2a2a;
+    background-color: #333;
     grid-area: footer;
 }
 
@@ -34,23 +34,49 @@ body {
 
 .Index-nodes {
     grid-area: main;
-    background: #2a2a2a;
 }
 
 .Index-node {
     margin: 5px;
     padding: 15px;
-    background: #1f1f1f;
+    background-color: #fff;
+    box-shadow: 0 0 5px rgba(0, 0, 0, 0.1);
+}
+
+span {
+    display: block;
+    margin-bottom: 5px;
+}
+
+input[type="text"] {
+    width: 100%;
+    padding: 10px;
+    border-radius: 5px;
+    border: 1px solid #ccc;
+    margin-bottom: 10px;
+}
+
+button {
+    background-color: #4c9aff;
+    color: #fff;
+    border: none;
+    border-radius: 5px;
+    padding: 10px 20px;
+    font-size: 1.2em;
+}
+
+button:hover {
+    background-color: #4c7eff;
 }
 
 a:link {
-    color: #50f020;
+    color: #333;
 }
 
 a:visited {
-    color: #50f020;
+    color: #333;
 }
 
 a:hover {
-    color: #30a00a;
+    color: #555;
 }

cmd/lotus-miner/actor.go

@@ -1166,7 +1166,7 @@ var actorConfirmChangeWorker = &cli.Command{
 var actorConfirmChangeBeneficiary = &cli.Command{
 	Name:      "confirm-change-beneficiary",
 	Usage:     "Confirm a beneficiary address change",
-	ArgsUsage: "[minerAddress]",
+	ArgsUsage: "[minerID]",
 	Flags: []cli.Flag{
 		&cli.BoolFlag{
 			Name:  "really-do-it",

cmd/lotus-miner/sectors.go

@@ -2,9 +2,11 @@ package main
 
 import (
 	"bufio"
+	"encoding/csv"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"math"
 	"os"
 	"sort"
 	"strconv"
@@ -318,6 +320,9 @@ var sectorsListCmd = &cli.Command{
 			Value: parallelSectorChecks,
 		},
 	},
+	Subcommands: []*cli.Command{
+		sectorsListUpgradeBoundsCmd,
+	},
 	Action: func(cctx *cli.Context) error {
 		// http mode allows for parallel json decoding/encoding, which was a bottleneck here
 		minerApi, closer, err := lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
@@ -585,6 +590,169 @@ var sectorsListCmd = &cli.Command{
 	},
 }
 
+var sectorsListUpgradeBoundsCmd = &cli.Command{
+	Name:  "upgrade-bounds",
+	Usage: "Output upgrade bounds for available sectors",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "buckets",
+			Value: 25,
+		},
+		&cli.BoolFlag{
+			Name:  "csv",
+			Usage: "output machine-readable values",
+		},
+		&cli.BoolFlag{
+			Name:  "deal-terms",
+			Usage: "bucket by how many deal-sectors can start at a given expiration",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		minerApi, closer, err := lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		fullApi, closer2, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer2()
+
+		ctx := lcli.ReqContext(cctx)
+
+		list, err := minerApi.SectorsListInStates(ctx, []api.SectorState{
+			api.SectorState(sealing.Available),
+		})
+		if err != nil {
+			return xerrors.Errorf("getting sector list: %w", err)
+		}
+
+		head, err := fullApi.ChainHead(ctx)
+		if err != nil {
+			return xerrors.Errorf("getting chain head: %w", err)
+		}
+
+		filter := bitfield.New()
+
+		for _, s := range list {
+			filter.Set(uint64(s))
+		}
+
+		maddr, err := minerApi.ActorAddress(ctx)
+		if err != nil {
+			return err
+		}
+
+		sset, err := fullApi.StateMinerSectors(ctx, maddr, &filter, head.Key())
+		if err != nil {
+			return err
+		}
+
+		if len(sset) == 0 {
+			return nil
+		}
+
+		var minExpiration, maxExpiration abi.ChainEpoch
+
+		for _, s := range sset {
+			if s.Expiration < minExpiration || minExpiration == 0 {
+				minExpiration = s.Expiration
+			}
+			if s.Expiration > maxExpiration {
+				maxExpiration = s.Expiration
+			}
+		}
+
+		buckets := cctx.Int("buckets")
+		bucketSize := (maxExpiration - minExpiration) / abi.ChainEpoch(buckets)
+		bucketCounts := make([]int, buckets+1)
+
+		for b := range bucketCounts {
+			bucketMin := minExpiration + abi.ChainEpoch(b)*bucketSize
+			bucketMax := minExpiration + abi.ChainEpoch(b+1)*bucketSize
+
+			if cctx.Bool("deal-terms") {
+				bucketMax = bucketMax + policy.MarketDefaultAllocationTermBuffer
+			}
+
+			for _, s := range sset {
+				isInBucket := s.Expiration >= bucketMin && s.Expiration < bucketMax
+
+				if isInBucket {
+					bucketCounts[b]++
+				}
+			}
+
+		}
+
+		// Creating CSV writer
+		writer := csv.NewWriter(os.Stdout)
+
+		// Writing CSV headers
+		err = writer.Write([]string{"Max Expiration in Bucket", "Sector Count"})
+		if err != nil {
+			return xerrors.Errorf("writing csv headers: %w", err)
+		}
+
+		// Writing bucket details
+
+		if cctx.Bool("csv") {
+			for i := 0; i < buckets; i++ {
+				maxExp := minExpiration + abi.ChainEpoch(i+1)*bucketSize
+
+				timeStr := strconv.FormatInt(int64(maxExp), 10)
+
+				err = writer.Write([]string{
+					timeStr,
+					strconv.Itoa(bucketCounts[i]),
+				})
+				if err != nil {
+					return xerrors.Errorf("writing csv row: %w", err)
+				}
+			}
+
+			// Flush to make sure all data is written to the underlying writer
+			writer.Flush()
+
+			if err := writer.Error(); err != nil {
+				return xerrors.Errorf("flushing csv writer: %w", err)
+			}
+
+			return nil
+		}
+
+		tw := tablewriter.New(
+			tablewriter.Col("Bucket Expiration"),
+			tablewriter.Col("Sector Count"),
+			tablewriter.Col("Bar"),
+		)
+
+		var barCols = 40
+		var maxCount int
+
+		for _, c := range bucketCounts {
+			if c > maxCount {
+				maxCount = c
+			}
+		}
+
+		for i := 0; i < buckets; i++ {
+			maxExp := minExpiration + abi.ChainEpoch(i+1)*bucketSize
+			timeStr := cliutil.EpochTime(head.Height(), maxExp)
+
+			tw.Write(map[string]interface{}{
+				"Bucket Expiration": timeStr,
+				"Sector Count":      color.YellowString("%d", bucketCounts[i]),
+				"Bar":               "[" + color.GreenString(strings.Repeat("|", bucketCounts[i]*barCols/maxCount)) + strings.Repeat(" ", barCols-bucketCounts[i]*barCols/maxCount) + "]",
+			})
+		}
+
+		return tw.Flush(os.Stdout)
+	},
+}
+
 var sectorsRefsCmd = &cli.Command{
 	Name:  "refs",
 	Usage: "List References to sectors",
@@ -1922,10 +2090,31 @@ var sectorsBatchingPendingCommit = &cli.Command{
 			for _, sector := range pending {
 				fmt.Println(sector.Number)
 			}
-			return nil
-		}
 
-		fmt.Println("No sectors queued to be committed")
+			reader := bufio.NewReader(os.Stdin)
+			fmt.Print("Do you want to publish these sectors now? (yes/no): ")
+			userInput, err := reader.ReadString('\n')
+			if err != nil {
+				return xerrors.Errorf("reading user input: %w", err)
+			}
+			userInput = strings.ToLower(strings.TrimSpace(userInput))
+
+			if userInput == "yes" {
+				err := cctx.Set("publish-now", "true")
+				if err != nil {
+					return xerrors.Errorf("setting publish-now flag: %w", err)
+				}
+				return cctx.Command.Action(cctx)
+			} else if userInput == "no" {
+				return nil
+			} else {
+				fmt.Println("Invalid input. Please answer with 'yes' or 'no'.")
+				return nil
+			}
+
+		} else {
+			fmt.Println("No sectors queued to be committed")
+		}
 		return nil
 	},
 }
@@ -1980,10 +2169,31 @@ var sectorsBatchingPendingPreCommit = &cli.Command{
 			for _, sector := range pending {
 				fmt.Println(sector.Number)
 			}
-			return nil
-		}
 
-		fmt.Println("No sectors queued to be committed")
+			reader := bufio.NewReader(os.Stdin)
+			fmt.Print("Do you want to publish these sectors now? (yes/no): ")
+			userInput, err := reader.ReadString('\n')
+			if err != nil {
+				return xerrors.Errorf("reading user input: %w", err)
+			}
+			userInput = strings.ToLower(strings.TrimSpace(userInput))
+
+			if userInput == "yes" {
+				err := cctx.Set("publish-now", "true")
+				if err != nil {
+					return xerrors.Errorf("setting publish-now flag: %w", err)
+				}
+				return cctx.Command.Action(cctx)
+			} else if userInput == "no" {
+				return nil
+			} else {
+				fmt.Println("Invalid input. Please answer with 'yes' or 'no'.")
+				return nil
+			}
+
+		} else {
+			fmt.Println("No sectors queued to be committed")
+		}
 		return nil
 	},
 }
@@ -2014,7 +2224,6 @@ func yesno(b bool) string {
 	return color.RedString("NO")
 }
 
-// TODO simulate this call if --really-do-it is not used
 var sectorsCompactPartitionsCmd = &cli.Command{
 	Name:  "compact-partitions",
 	Usage: "removes dead sectors from partitions and reduces the number of partitions used if possible",
@@ -2040,12 +2249,7 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 		},
 	},
 	Action: func(cctx *cli.Context) error {
-		if !cctx.Bool("really-do-it") {
-			fmt.Println("Pass --really-do-it to actually execute this action")
-			return nil
-		}
-
-		api, acloser, err := lcli.GetFullNodeAPI(cctx)
+		fullNodeAPI, acloser, err := lcli.GetFullNodeAPI(cctx)
 		if err != nil {
 			return err
 		}
@@ -2058,7 +2262,7 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 			return err
 		}
 
-		minfo, err := api.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+		minfo, err := fullNodeAPI.StateMinerInfo(ctx, maddr, types.EmptyTSK)
 		if err != nil {
 			return err
 		}
@@ -2074,46 +2278,118 @@ var sectorsCompactPartitionsCmd = &cli.Command{
 		}
 		fmt.Printf("compacting %d paritions\n", len(parts))
 
+		var makeMsgForPartitions func(partitionsBf bitfield.BitField) ([]*types.Message, error)
+		makeMsgForPartitions = func(partitionsBf bitfield.BitField) ([]*types.Message, error) {
+			params := miner.CompactPartitionsParams{
+				Deadline:   deadline,
+				Partitions: partitionsBf,
+			}
+
+			sp, aerr := actors.SerializeParams(&params)
+			if aerr != nil {
+				return nil, xerrors.Errorf("serializing params: %w", err)
+			}
+
+			msg := &types.Message{
+				From:   minfo.Worker,
+				To:     maddr,
+				Method: builtin.MethodsMiner.CompactPartitions,
+				Value:  big.Zero(),
+				Params: sp,
+			}
+
+			estimatedMsg, err := fullNodeAPI.GasEstimateMessageGas(ctx, msg, nil, types.EmptyTSK)
+			if err != nil && xerrors.Is(err, &api.ErrOutOfGas{}) {
+				// the message is too big -- split into 2
+				partitionsSlice, err := partitionsBf.All(math.MaxUint64)
+				if err != nil {
+					return nil, err
+				}
+
+				partitions1 := bitfield.New()
+				for i := 0; i < len(partitionsSlice)/2; i++ {
+					partitions1.Set(uint64(i))
+				}
+
+				msgs1, err := makeMsgForPartitions(partitions1)
+				if err != nil {
+					return nil, err
+				}
+
+				// time for the second half
+				partitions2 := bitfield.New()
+				for i := len(partitionsSlice) / 2; i < len(partitionsSlice); i++ {
+					partitions2.Set(uint64(i))
+				}
+
+				msgs2, err := makeMsgForPartitions(partitions2)
+				if err != nil {
+					return nil, err
+				}
+
+				return append(msgs1, msgs2...), nil
+			} else if err != nil {
+				return nil, err
+			}
+
+			return []*types.Message{estimatedMsg}, nil
+		}
+
 		partitions := bitfield.New()
 		for _, partition := range parts {
 			partitions.Set(uint64(partition))
 		}
 
-		params := miner.CompactPartitionsParams{
-			Deadline:   deadline,
-			Partitions: partitions,
-		}
-
-		sp, err := actors.SerializeParams(&params)
+		msgs, err := makeMsgForPartitions(partitions)
 		if err != nil {
-			return xerrors.Errorf("serializing params: %w", err)
+			return xerrors.Errorf("failed to make messages: %w", err)
 		}
 
-		smsg, err := api.MpoolPushMessage(ctx, &types.Message{
-			From:   minfo.Worker,
-			To:     maddr,
-			Method: builtin.MethodsMiner.CompactPartitions,
-			Value:  big.Zero(),
-			Params: sp,
-		}, nil)
-		if err != nil {
-			return xerrors.Errorf("mpool push: %w", err)
+		// Actually send the messages if really-do-it provided, simulate otherwise
+		if cctx.Bool("really-do-it") {
+			smsgs, err := fullNodeAPI.MpoolBatchPushMessage(ctx, msgs, nil)
+			if err != nil {
+				return xerrors.Errorf("mpool push: %w", err)
+			}
+
+			if len(smsgs) == 1 {
+				fmt.Printf("Requested compact partitions in message %s\n", smsgs[0].Cid())
+			} else {
+				fmt.Printf("Requested compact partitions in %d messages\n\n", len(smsgs))
+				for _, v := range smsgs {
+					fmt.Println(v.Cid())
+				}
+			}
+
+			for _, v := range smsgs {
+				wait, err := fullNodeAPI.StateWaitMsg(ctx, v.Cid(), 2)
+				if err != nil {
+					return err
+				}
+
+				// check it executed successfully
+				if wait.Receipt.ExitCode.IsError() {
+					fmt.Println(cctx.App.Writer, "compact partitions msg %s failed!", v.Cid())
+					return err
+				}
+			}
+
+			return nil
 		}
 
-		fmt.Printf("Requested compact partitions in message %s\n", smsg.Cid())
+		for i, v := range msgs {
+			fmt.Printf("total of %d CompactPartitions msgs would be sent\n", len(msgs))
 
-		wait, err := api.StateWaitMsg(ctx, smsg.Cid(), 0)
-		if err != nil {
-			return err
-		}
+			estMsg, err := fullNodeAPI.GasEstimateMessageGas(ctx, v, nil, types.EmptyTSK)
+			if err != nil {
+				return err
+			}
 
-		// check it executed successfully
-		if wait.Receipt.ExitCode.IsError() {
-			fmt.Println(cctx.App.Writer, "compact partitions failed!")
-			return err
+			fmt.Printf("msg %d would cost up to %s\n", i+1, types.FIL(estMsg.RequiredFunds()))
 		}
 
 		return nil
+
 	},
 }
 

cmd/lotus-miner/storage.go

@@ -225,6 +225,7 @@ var storageRedeclareCmd = &cli.Command{
 		&cli.BoolFlag{
 			Name:  "drop-missing",
 			Usage: "Drop index entries with missing files",
+			Value: true,
 		},
 	},
 	Action: func(cctx *cli.Context) error {
@@ -235,14 +236,19 @@ var storageRedeclareCmd = &cli.Command{
 		defer closer()
 		ctx := lcli.ReqContext(cctx)
 
-		if cctx.NArg() != 1 {
-			return lcli.IncorrectNumArgs(cctx)
+		// check if no argument and no --id or --all flag is provided
+		if cctx.NArg() == 0 && !cctx.IsSet("id") && !cctx.Bool("all") {
+			return xerrors.Errorf("You must specify a storage path, or --id, or --all")
 		}
 
 		if cctx.IsSet("id") && cctx.Bool("all") {
 			return xerrors.Errorf("--id and --all can't be passed at the same time")
 		}
 
+		if cctx.Bool("all") && cctx.NArg() > 0 {
+			return xerrors.Errorf("No additional arguments are expected when --all is set")
+		}
+
 		if cctx.IsSet("id") {
 			id := storiface.ID(cctx.String("id"))
 			return minerApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
@@ -252,7 +258,28 @@ var storageRedeclareCmd = &cli.Command{
 			return minerApi.StorageRedeclareLocal(ctx, nil, cctx.Bool("drop-missing"))
 		}
 
-		return xerrors.Errorf("either --all or --id must be specified")
+		// As no --id or --all flag is set, we can assume the argument is a path.
+		path := cctx.Args().First()
+		metaFilePath := filepath.Join(path, "sectorstore.json")
+
+		var meta storiface.LocalStorageMeta
+		metaFile, err := os.Open(metaFilePath)
+		if err != nil {
+			return xerrors.Errorf("Failed to open file: %w", err)
+		}
+		defer func() {
+			if closeErr := metaFile.Close(); closeErr != nil {
+				log.Error("Failed to close the file: %v", closeErr)
+			}
+		}()
+
+		err = json.NewDecoder(metaFile).Decode(&meta)
+		if err != nil {
+			return xerrors.Errorf("Failed to decode file: %w", err)
+		}
+
+		id := meta.ID
+		return minerApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
 	},
 }
 

cmd/lotus-shed/actor.go

@@ -968,7 +968,7 @@ var actorProposeChangeBeneficiary = &cli.Command{
 var actorConfirmChangeBeneficiary = &cli.Command{
 	Name:      "confirm-change-beneficiary",
 	Usage:     "Confirm a beneficiary address change",
-	ArgsUsage: "[minerAddress]",
+	ArgsUsage: "[minerID]",
 	Flags: []cli.Flag{
 		&cli.BoolFlag{
 			Name:  "really-do-it",

cmd/lotus-shed/eth.go

@@ -9,6 +9,7 @@ import (
 	"github.com/filecoin-project/go-state-types/abi"
 
 	"github.com/filecoin-project/lotus/chain/types"
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
 	lcli "github.com/filecoin-project/lotus/cli"
 )
 
@@ -23,6 +24,7 @@ var ethCmd = &cli.Command{
 	},
 	Subcommands: []*cli.Command{
 		checkTipsetsCmd,
+		computeEthHashCmd,
 	},
 }
 
@@ -70,3 +72,36 @@ var checkTipsetsCmd = &cli.Command{
 		return nil
 	},
 }
+
+var computeEthHashCmd = &cli.Command{
+	Name:  "compute-eth-hash",
+	Usage: "Compute the eth hash for a given message CID",
+	Action: func(cctx *cli.Context) error {
+		if cctx.NArg() != 1 {
+			return lcli.IncorrectNumArgs(cctx)
+		}
+
+		msg, err := messageFromString(cctx, cctx.Args().First())
+		if err != nil {
+			return err
+		}
+
+		switch msg := msg.(type) {
+		case *types.SignedMessage:
+			tx, err := ethtypes.EthTxFromSignedEthMessage(msg)
+			if err != nil {
+				return fmt.Errorf("failed to convert from signed message: %w", err)
+			}
+
+			tx.Hash, err = tx.TxHash()
+			if err != nil {
+				return fmt.Errorf("failed to call TxHash: %w", err)
+			}
+			fmt.Println(tx.Hash)
+		default:
+			return fmt.Errorf("not a signed message")
+		}
+
+		return nil
+	},
+}

cmd/lotus-shed/indexes.go

@@ -0,0 +1,337 @@
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"path"
+	"path/filepath"
+	"strings"
+
+	"github.com/mitchellh/go-homedir"
+	"github.com/urfave/cli/v2"
+	"golang.org/x/xerrors"
+
+	"github.com/filecoin-project/go-state-types/abi"
+	"github.com/filecoin-project/go-state-types/crypto"
+
+	"github.com/filecoin-project/lotus/chain/types/ethtypes"
+	lcli "github.com/filecoin-project/lotus/cli"
+)
+
+func withCategory(cat string, cmd *cli.Command) *cli.Command {
+	cmd.Category = strings.ToUpper(cat)
+	return cmd
+}
+
+var indexesCmd = &cli.Command{
+	Name:            "indexes",
+	Usage:           "Commands related to managing sqlite indexes",
+	HideHelpCommand: true,
+	Subcommands: []*cli.Command{
+		withCategory("msgindex", backfillMsgIndexCmd),
+		withCategory("msgindex", pruneMsgIndexCmd),
+		withCategory("txhash", backfillTxHashCmd),
+	},
+}
+
+var backfillMsgIndexCmd = &cli.Command{
+	Name:  "backfill-msgindex",
+	Usage: "Backfill the msgindex.db for a number of epochs starting from a specified height",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "from",
+			Value: 0,
+			Usage: "height to start the backfill; uses the current head if omitted",
+		},
+		&cli.IntFlag{
+			Name:  "epochs",
+			Value: 1800,
+			Usage: "number of epochs to backfill; defaults to 1800 (2 finalities)",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+
+		defer closer()
+		ctx := lcli.ReqContext(cctx)
+
+		curTs, err := api.ChainHead(ctx)
+		if err != nil {
+			return err
+		}
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight == 0 {
+			startHeight = int64(curTs.Height()) - 1
+		}
+		epochs := cctx.Int("epochs")
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		insertStmt, err := db.Prepare("INSERT OR IGNORE INTO messages (cid, tipset_cid, epoch) VALUES (?, ?, ?)")
+		if err != nil {
+			return err
+		}
+
+		var nrRowsAffected int64
+		for i := 0; i < epochs; i++ {
+			epoch := abi.ChainEpoch(startHeight - int64(i))
+
+			if i%100 == 0 {
+				log.Infof("%d/%d processing epoch:%d, nrRowsAffected:%d", i, epochs, epoch, nrRowsAffected)
+			}
+
+			ts, err := api.ChainGetTipSetByHeight(ctx, epoch, curTs.Key())
+			if err != nil {
+				return fmt.Errorf("failed to get tipset at epoch %d: %w", epoch, err)
+			}
+
+			tsCid, err := ts.Key().Cid()
+			if err != nil {
+				return fmt.Errorf("failed to get tipset cid at epoch %d: %w", epoch, err)
+			}
+
+			msgs, err := api.ChainGetMessagesInTipset(ctx, ts.Key())
+			if err != nil {
+				return fmt.Errorf("failed to get messages in tipset at epoch %d: %w", epoch, err)
+			}
+
+			for _, msg := range msgs {
+				key := msg.Cid.String()
+				tskey := tsCid.String()
+				res, err := insertStmt.Exec(key, tskey, int64(epoch))
+				if err != nil {
+					return fmt.Errorf("failed to insert message cid %s in tipset %s at epoch %d: %w", key, tskey, epoch, err)
+				}
+				rowsAffected, err := res.RowsAffected()
+				if err != nil {
+					return fmt.Errorf("failed to get rows affected for message cid %s in tipset %s at epoch %d: %w", key, tskey, epoch, err)
+				}
+				nrRowsAffected += rowsAffected
+			}
+		}
+
+		log.Infof("Done backfilling, nrRowsAffected:%d", nrRowsAffected)
+
+		return nil
+	},
+}
+
+var pruneMsgIndexCmd = &cli.Command{
+	Name:  "prune-msgindex",
+	Usage: "Prune the msgindex.db for messages included before a given epoch",
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:  "from",
+			Usage: "height to start the prune; if negative it indicates epochs from current head",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+
+		defer closer()
+		ctx := lcli.ReqContext(cctx)
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight < 0 {
+			curTs, err := api.ChainHead(ctx)
+			if err != nil {
+				return err
+			}
+
+			startHeight += int64(curTs.Height())
+
+			if startHeight < 0 {
+				return xerrors.Errorf("bogus start height %d", startHeight)
+			}
+		}
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		tx, err := db.Begin()
+		if err != nil {
+			return err
+		}
+
+		if _, err := tx.Exec("DELETE FROM messages WHERE epoch < ?", startHeight); err != nil {
+			if err := tx.Rollback(); err != nil {
+				fmt.Printf("ERROR: rollback: %s", err)
+			}
+			return err
+		}
+
+		if err := tx.Commit(); err != nil {
+			return err
+		}
+
+		return nil
+	},
+}
+
+var backfillTxHashCmd = &cli.Command{
+	Name:  "backfill-txhash",
+	Usage: "Backfills the txhash.db for a number of epochs starting from a specified height",
+	Flags: []cli.Flag{
+		&cli.UintFlag{
+			Name:  "from",
+			Value: 0,
+			Usage: "the tipset height to start backfilling from (0 is head of chain)",
+		},
+		&cli.IntFlag{
+			Name:  "epochs",
+			Value: 2000,
+			Usage: "the number of epochs to backfill",
+		},
+	},
+	Action: func(cctx *cli.Context) error {
+		api, closer, err := lcli.GetFullNodeAPI(cctx)
+		if err != nil {
+			return err
+		}
+		defer closer()
+
+		ctx := lcli.ReqContext(cctx)
+
+		curTs, err := api.ChainHead(ctx)
+		if err != nil {
+			return err
+		}
+
+		startHeight := int64(cctx.Int("from"))
+		if startHeight == 0 {
+			startHeight = int64(curTs.Height()) - 1
+		}
+
+		epochs := cctx.Int("epochs")
+
+		basePath, err := homedir.Expand(cctx.String("repo"))
+		if err != nil {
+			return err
+		}
+
+		dbPath := filepath.Join(basePath, "sqlite", "txhash.db")
+		db, err := sql.Open("sqlite3", dbPath)
+		if err != nil {
+			return err
+		}
+
+		defer func() {
+			err := db.Close()
+			if err != nil {
+				fmt.Printf("ERROR: closing db: %s", err)
+			}
+		}()
+
+		insertStmt, err := db.Prepare("INSERT OR IGNORE INTO eth_tx_hashes(hash, cid) VALUES(?, ?)")
+		if err != nil {
+			return err
+		}
+
+		var totalRowsAffected int64 = 0
+		for i := 0; i < epochs; i++ {
+			epoch := abi.ChainEpoch(startHeight - int64(i))
+
+			select {
+			case <-cctx.Done():
+				fmt.Println("request cancelled")
+				return nil
+			default:
+			}
+
+			curTsk := curTs.Parents()
+			execTs, err := api.ChainGetTipSet(ctx, curTsk)
+			if err != nil {
+				return fmt.Errorf("failed to call ChainGetTipSet for %s: %w", curTsk, err)
+			}
+
+			if i%100 == 0 {
+				log.Infof("%d/%d processing epoch:%d", i, epochs, epoch)
+			}
+
+			for _, blockheader := range execTs.Blocks() {
+				blkMsgs, err := api.ChainGetBlockMessages(ctx, blockheader.Cid())
+				if err != nil {
+					log.Infof("Could not get block messages at epoch: %d, stopping walking up the chain", epoch)
+					epochs = i
+					break
+				}
+
+				for _, smsg := range blkMsgs.SecpkMessages {
+					if smsg.Signature.Type != crypto.SigTypeDelegated {
+						continue
+					}
+
+					tx, err := ethtypes.EthTxFromSignedEthMessage(smsg)
+					if err != nil {
+						return fmt.Errorf("failed to convert from signed message: %w at epoch: %d", err, epoch)
+					}
+
+					tx.Hash, err = tx.TxHash()
+					if err != nil {
+						return fmt.Errorf("failed to calculate hash for ethTx: %w at epoch: %d", err, epoch)
+					}
+
+					res, err := insertStmt.Exec(tx.Hash.String(), smsg.Cid().String())
+					if err != nil {
+						return fmt.Errorf("error inserting tx mapping to db: %s at epoch: %d", err, epoch)
+					}
+
+					rowsAffected, err := res.RowsAffected()
+					if err != nil {
+						return fmt.Errorf("error getting rows affected: %s at epoch: %d", err, epoch)
+					}
+
+					if rowsAffected > 0 {
+						log.Debugf("Inserted txhash %s, cid: %s at epoch: %d", tx.Hash.String(), smsg.Cid().String(), epoch)
+					}
+
+					totalRowsAffected += rowsAffected
+				}
+			}
+
+			curTs = execTs
+		}
+
+		log.Infof("Done, inserted %d missing txhashes", totalRowsAffected)
+
+		return nil
+	},
+}

cmd/lotus-shed/main.go

@@ -1,8 +1,10 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
+	"os/signal"
 
 	logging "github.com/ipfs/go-log/v2"
 	"github.com/urfave/cli/v2"
@@ -84,7 +86,7 @@ func main() {
 		invariantsCmd,
 		gasTraceCmd,
 		replayOfflineCmd,
-		msgindexCmd,
+		indexesCmd,
 		FevmAnalyticsCmd,
 		mismatchesCmd,
 	}
@@ -118,7 +120,20 @@ func main() {
 		},
 	}
 
-	if err := app.Run(os.Args); err != nil {
+	// terminate early on ctrl+c
+	c := make(chan os.Signal, 1)
+	signal.Notify(c, os.Interrupt)
+	ctx, cancel := context.WithCancel(context.Background())
+	go func() {
+		<-c
+		cancel()
+		fmt.Println("Received interrupt, shutting down... Press CTRL+C again to force shutdown")
+		<-c
+		fmt.Println("Forcing stop")
+		os.Exit(1)
+	}()
+
+	if err := app.RunContext(ctx, os.Args); err != nil {
 		log.Errorf("%+v", err)
 		os.Exit(1)
 		return

cmd/lotus-shed/msgindex.go

@@ -1,221 +0,0 @@
-package main
-
-import (
-	"database/sql"
-	"fmt"
-	"path"
-
-	"github.com/ipfs/go-cid"
-	_ "github.com/mattn/go-sqlite3"
-	"github.com/mitchellh/go-homedir"
-	"github.com/urfave/cli/v2"
-	"golang.org/x/xerrors"
-
-	"github.com/filecoin-project/go-state-types/abi"
-
-	lcli "github.com/filecoin-project/lotus/cli"
-)
-
-var msgindexCmd = &cli.Command{
-	Name:  "msgindex",
-	Usage: "Tools for managing the message index",
-	Subcommands: []*cli.Command{
-		msgindexBackfillCmd,
-		msgindexPruneCmd,
-	},
-}
-
-var msgindexBackfillCmd = &cli.Command{
-	Name:  "backfill",
-	Usage: "Backfill the message index for a number of epochs starting from a specified height",
-	Flags: []cli.Flag{
-		&cli.IntFlag{
-			Name:  "from",
-			Value: 0,
-			Usage: "height to start the backfill; uses the current head if omitted",
-		},
-		&cli.IntFlag{
-			Name:  "epochs",
-			Value: 1800,
-			Usage: "number of epochs to backfill; defaults to 1800 (2 finalities)",
-		},
-		&cli.StringFlag{
-			Name:  "repo",
-			Value: "~/.lotus",
-			Usage: "path to the repo",
-		},
-	},
-	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetFullNodeAPI(cctx)
-		if err != nil {
-			return err
-		}
-
-		defer closer()
-		ctx := lcli.ReqContext(cctx)
-
-		curTs, err := api.ChainHead(ctx)
-		if err != nil {
-			return err
-		}
-
-		startHeight := int64(cctx.Int("from"))
-		if startHeight == 0 {
-			startHeight = int64(curTs.Height()) - 1
-		}
-		epochs := cctx.Int("epochs")
-
-		basePath, err := homedir.Expand(cctx.String("repo"))
-		if err != nil {
-			return err
-		}
-
-		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
-		db, err := sql.Open("sqlite3", dbPath)
-		if err != nil {
-			return err
-		}
-
-		defer func() {
-			err := db.Close()
-			if err != nil {
-				fmt.Printf("ERROR: closing db: %s", err)
-			}
-		}()
-
-		tx, err := db.Begin()
-		if err != nil {
-			return err
-		}
-
-		insertStmt, err := tx.Prepare("INSERT INTO messages VALUES (?, ?, ?)")
-		if err != nil {
-			return err
-		}
-
-		insertMsg := func(cid, tsCid cid.Cid, epoch abi.ChainEpoch) error {
-			key := cid.String()
-			tskey := tsCid.String()
-			if _, err := insertStmt.Exec(key, tskey, int64(epoch)); err != nil {
-				return err
-			}
-
-			return nil
-		}
-		rollback := func() {
-			if err := tx.Rollback(); err != nil {
-				fmt.Printf("ERROR: rollback: %s", err)
-			}
-		}
-
-		for i := 0; i < epochs; i++ {
-			epoch := abi.ChainEpoch(startHeight - int64(i))
-
-			ts, err := api.ChainGetTipSetByHeight(ctx, epoch, curTs.Key())
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			tsCid, err := ts.Key().Cid()
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			msgs, err := api.ChainGetMessagesInTipset(ctx, ts.Key())
-			if err != nil {
-				rollback()
-				return err
-			}
-
-			for _, msg := range msgs {
-				if err := insertMsg(msg.Cid, tsCid, epoch); err != nil {
-					rollback()
-					return err
-				}
-			}
-		}
-
-		if err := tx.Commit(); err != nil {
-			return err
-		}
-
-		return nil
-	},
-}
-
-var msgindexPruneCmd = &cli.Command{
-	Name:  "prune",
-	Usage: "Prune the message index for messages included before a given epoch",
-	Flags: []cli.Flag{
-		&cli.IntFlag{
-			Name:  "from",
-			Usage: "height to start the prune; if negative it indicates epochs from current head",
-		},
-		&cli.StringFlag{
-			Name:  "repo",
-			Value: "~/.lotus",
-			Usage: "path to the repo",
-		},
-	},
-	Action: func(cctx *cli.Context) error {
-		api, closer, err := lcli.GetFullNodeAPI(cctx)
-		if err != nil {
-			return err
-		}
-
-		defer closer()
-		ctx := lcli.ReqContext(cctx)
-
-		startHeight := int64(cctx.Int("from"))
-		if startHeight < 0 {
-			curTs, err := api.ChainHead(ctx)
-			if err != nil {
-				return err
-			}
-
-			startHeight += int64(curTs.Height())
-
-			if startHeight < 0 {
-				return xerrors.Errorf("bogus start height %d", startHeight)
-			}
-		}
-
-		basePath, err := homedir.Expand(cctx.String("repo"))
-		if err != nil {
-			return err
-		}
-
-		dbPath := path.Join(basePath, "sqlite", "msgindex.db")
-		db, err := sql.Open("sqlite3", dbPath)
-		if err != nil {
-			return err
-		}
-
-		defer func() {
-			err := db.Close()
-			if err != nil {
-				fmt.Printf("ERROR: closing db: %s", err)
-			}
-		}()
-
-		tx, err := db.Begin()
-		if err != nil {
-			return err
-		}
-
-		if _, err := tx.Exec("DELETE FROM messages WHERE epoch < ?", startHeight); err != nil {
-			if err := tx.Rollback(); err != nil {
-				fmt.Printf("ERROR: rollback: %s", err)
-			}
-			return err
-		}
-
-		if err := tx.Commit(); err != nil {
-			return err
-		}
-
-		return nil
-	},
-}

cmd/lotus-shed/sectors.go

@@ -653,7 +653,7 @@ fr32 padding is removed from the output.`,
 			return xerrors.Errorf("getting reader: %w", err)
 		}
 
-		rd, err := readStarter(0)
+		rd, err := readStarter(0, storiface.PaddedByteIndex(length))
 		if err != nil {
 			return xerrors.Errorf("starting reader: %w", err)
 		}

cmd/lotus-shed/verifreg.go

@@ -474,7 +474,7 @@ var verifRegRemoveVerifiedClientDataCapCmd = &cli.Command{
 
 		st, err := multisig.Load(store, vrkState)
 		if err != nil {
-			return err
+			return fmt.Errorf("load vrk failed: %w ", err)
 		}
 
 		signers, err := st.Signers()
@@ -508,14 +508,13 @@ var verifRegRemoveVerifiedClientDataCapCmd = &cli.Command{
 			return err
 		}
 
-		sm, _, err := srv.PublishMessage(ctx, proto, false)
+		sm, err := lcli.InteractiveSend(ctx, cctx, srv, proto)
 		if err != nil {
 			return err
 		}
 
 		msgCid := sm.Cid()
-
-		fmt.Printf("message sent, now waiting on cid: %s\n", msgCid)
+		fmt.Println("sending msg: ", msgCid)
 
 		mwait, err := api.StateWaitMsg(ctx, msgCid, uint64(cctx.Int("confidence")), build.Finality, true)
 		if err != nil {

cmd/lotus-worker/main.go

@@ -275,6 +275,12 @@ var runCmd = &cli.Command{
 			Name:  "http-server-timeout",
 			Value: "30s",
 		},
+		&cli.BoolFlag{
+			Name:        "data-cid",
+			Usage:       "Run the data-cid task. true|false",
+			Value:       true,
+			DefaultText: "inherits --addpiece",
+		},
 	},
 	Before: func(cctx *cli.Context) error {
 		if cctx.IsSet("address") {
@@ -386,8 +392,19 @@ var runCmd = &cli.Command{
 			}
 		}
 
+		ttDataCidDefault := false
 		if (workerType == sealtasks.WorkerSealing || cctx.IsSet("addpiece")) && cctx.Bool("addpiece") {
-			taskTypes = append(taskTypes, sealtasks.TTAddPiece, sealtasks.TTDataCid)
+			taskTypes = append(taskTypes, sealtasks.TTAddPiece)
+			ttDataCidDefault = true
+		}
+		if workerType == sealtasks.WorkerSealing {
+			if cctx.IsSet("data-cid") {
+				if cctx.Bool("data-cid") {
+					taskTypes = append(taskTypes, sealtasks.TTDataCid)
+				}
+			} else if ttDataCidDefault {
+				taskTypes = append(taskTypes, sealtasks.TTDataCid)
+			}
 		}
 		if (workerType == sealtasks.WorkerSealing || cctx.IsSet("sector-download")) && cctx.Bool("sector-download") {
 			taskTypes = append(taskTypes, sealtasks.TTDownloadSector)

cmd/lotus-worker/storage.go

@@ -178,6 +178,7 @@ var storageRedeclareCmd = &cli.Command{
 		&cli.BoolFlag{
 			Name:  "drop-missing",
 			Usage: "Drop index entries with missing files",
+			Value: true,
 		},
 	},
 	Action: func(cctx *cli.Context) error {
@@ -188,10 +189,19 @@ var storageRedeclareCmd = &cli.Command{
 		defer closer()
 		ctx := lcli.ReqContext(cctx)
 
+		// check if no argument and no --id or --all flag is provided
+		if cctx.NArg() == 0 && !cctx.IsSet("id") && !cctx.Bool("all") {
+			return xerrors.Errorf("You must specify a storage path, or --id, or --all")
+		}
+
 		if cctx.IsSet("id") && cctx.Bool("all") {
 			return xerrors.Errorf("--id and --all can't be passed at the same time")
 		}
 
+		if cctx.Bool("all") && cctx.NArg() > 0 {
+			return xerrors.Errorf("No additional arguments are expected when --all is set")
+		}
+
 		if cctx.IsSet("id") {
 			id := storiface.ID(cctx.String("id"))
 			return nodeApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
@@ -201,6 +211,27 @@ var storageRedeclareCmd = &cli.Command{
 			return nodeApi.StorageRedeclareLocal(ctx, nil, cctx.Bool("drop-missing"))
 		}
 
-		return xerrors.Errorf("either --all or --id must be specified")
+		// As no --id or --all flag is set, we can assume the argument is a path.
+		path := cctx.Args().First()
+		metaFilePath := filepath.Join(path, "sectorstore.json")
+
+		var meta storiface.LocalStorageMeta
+		metaFile, err := os.Open(metaFilePath)
+		if err != nil {
+			return xerrors.Errorf("Failed to open file: %w", err)
+		}
+		defer func() {
+			if closeErr := metaFile.Close(); closeErr != nil {
+				log.Error("Failed to close the file: %v", closeErr)
+			}
+		}()
+
+		err = json.NewDecoder(metaFile).Decode(&meta)
+		if err != nil {
+			return xerrors.Errorf("Failed to decode file: %w", err)
+		}
+
+		id := meta.ID
+		return nodeApi.StorageRedeclareLocal(ctx, &id, cctx.Bool("drop-missing"))
 	},
 }

cmd/lotus/daemon.go

@@ -16,9 +16,12 @@ import (
 	"strings"
 
 	"github.com/DataDog/zstd"
+	"github.com/ipfs/go-cid"
+	levelds "github.com/ipfs/go-ds-leveldb"
 	metricsprom "github.com/ipfs/go-metrics-prometheus"
 	"github.com/mitchellh/go-homedir"
 	"github.com/multiformats/go-multiaddr"
+	ldbopts "github.com/syndtr/goleveldb/leveldb/opt"
 	"github.com/urfave/cli/v2"
 	"go.opencensus.io/plugin/runmetrics"
 	"go.opencensus.io/stats"
@@ -27,13 +30,19 @@ import (
 	"golang.org/x/xerrors"
 	"gopkg.in/cheggaaa/pb.v1"
 
+	"github.com/filecoin-project/go-address"
+	cborutil "github.com/filecoin-project/go-cbor-util"
 	"github.com/filecoin-project/go-jsonrpc"
 	"github.com/filecoin-project/go-paramfetch"
+	"github.com/filecoin-project/go-state-types/builtin"
+	"github.com/filecoin-project/specs-actors/actors/builtin/miner"
 
 	lapi "github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/chain/actors"
 	"github.com/filecoin-project/lotus/chain/consensus"
 	"github.com/filecoin-project/lotus/chain/consensus/filcns"
+	"github.com/filecoin-project/lotus/chain/gen/slashfilter"
 	"github.com/filecoin-project/lotus/chain/index"
 	"github.com/filecoin-project/lotus/chain/stmgr"
 	"github.com/filecoin-project/lotus/chain/store"
@@ -160,6 +169,19 @@ var DaemonCmd = &cli.Command{
 			Name:  "restore-config",
 			Usage: "config file to use when restoring from backup",
 		},
+		&cli.BoolFlag{
+			Name:  "slash-consensus",
+			Usage: "Report consensus fault",
+			Value: false,
+		},
+		&cli.StringFlag{
+			Name:  "slasher-sender",
+			Usage: "optionally specify the account to report consensus from",
+		},
+		&cli.StringFlag{
+			Name:  "slashdb-dir",
+			Value: "slash watch db dir path",
+		},
 	},
 	Action: func(cctx *cli.Context) error {
 		isLite := cctx.Bool("lite")
@@ -380,7 +402,14 @@ var DaemonCmd = &cli.Command{
 		if err != nil {
 			return fmt.Errorf("failed to start json-rpc endpoint: %s", err)
 		}
-
+		if cctx.IsSet("slash-consensus") && cctx.IsSet("slashdb-dir") {
+			go func() {
+				err := slashConsensus(api, cctx.String("slashdb-dir"), cctx.String("slasher-sender"))
+				if err != nil {
+					panic("slashConsensus error")
+				}
+			}()
+		}
 		// Monitor for shutdown.
 		finishCh := node.MonitorShutdown(shutdownChan,
 			node.ShutdownHandler{Component: "rpc server", StopFunc: rpcStopper},
@@ -574,3 +603,122 @@ func ImportChain(ctx context.Context, r repo.Repo, fname string, snapshot bool)
 
 	return nil
 }
+
+func slashConsensus(a lapi.FullNode, p string, from string) error {
+	ctx := context.Background()
+	var fromAddr address.Address
+
+	ds, err := levelds.NewDatastore(p, &levelds.Options{
+		Compression: ldbopts.NoCompression,
+		NoSync:      false,
+		Strict:      ldbopts.StrictAll,
+		ReadOnly:    false,
+	})
+	if err != nil {
+		return xerrors.Errorf("open leveldb: %w", err)
+	}
+	sf := slashfilter.New(ds)
+	if from == "" {
+		defaddr, err := a.WalletDefaultAddress(ctx)
+		if err != nil {
+			return err
+		}
+		fromAddr = defaddr
+	} else {
+		addr, err := address.NewFromString(from)
+		if err != nil {
+			return err
+		}
+
+		fromAddr = addr
+	}
+
+	blocks, err := a.SyncIncomingBlocks(ctx)
+	if err != nil {
+		return xerrors.Errorf("sync incoming blocks failed: %w", err)
+	}
+	for block := range blocks {
+		log.Infof("deal with block: %d, %v, %s", block.Height, block.Miner, block.Cid())
+		if otherBlock, extraBlock, err := slashFilterMinedBlock(ctx, sf, a, block); err != nil {
+			if otherBlock == nil {
+				continue
+			}
+			log.Errorf("<!!> SLASH FILTER ERROR: %s", err)
+			bh1, err := cborutil.Dump(otherBlock)
+			if err != nil {
+				log.Errorf("could not dump otherblock:%s, err:%s", otherBlock.Cid(), err)
+				continue
+			}
+
+			bh2, err := cborutil.Dump(block)
+			if err != nil {
+				log.Errorf("could not dump block:%s, err:%s", block.Cid(), err)
+				continue
+			}
+
+			params := miner.ReportConsensusFaultParams{
+				BlockHeader1: bh1,
+				BlockHeader2: bh2,
+			}
+			if extraBlock != nil {
+				be, err := cborutil.Dump(extraBlock)
+				if err != nil {
+					log.Errorf("could not dump block:%s, err:%s", block.Cid(), err)
+					continue
+				}
+				params.BlockHeaderExtra = be
+			}
+
+			enc, err := actors.SerializeParams(&params)
+			if err != nil {
+				log.Errorf("could not serialize declare faults parameters: %s", err)
+				continue
+			}
+			message, err := a.MpoolPushMessage(ctx, &types.Message{
+				To:     block.Miner,
+				From:   fromAddr,
+				Value:  types.NewInt(0),
+				Method: builtin.MethodsMiner.ReportConsensusFault,
+				Params: enc,
+			}, nil)
+			if err != nil {
+				log.Errorf("ReportConsensusFault to messagepool error:%w", err)
+				continue
+			}
+			log.Infof("ReportConsensusFault message CID:%s", message.Cid())
+
+		}
+	}
+	return err
+}
+
+func slashFilterMinedBlock(ctx context.Context, sf *slashfilter.SlashFilter, a lapi.FullNode, blockB *types.BlockHeader) (*types.BlockHeader, *types.BlockHeader, error) {
+	blockC, err := a.ChainGetBlock(ctx, blockB.Parents[0])
+	if err != nil {
+		return nil, nil, xerrors.Errorf("chain get block error:%s", err)
+	}
+	otherCid, err := sf.MinedBlock(ctx, blockB, blockC.Height)
+	if err != nil {
+		return nil, nil, xerrors.Errorf("slash filter check block error:%s", err)
+	}
+	if otherCid != cid.Undef {
+		otherHeader, err := a.ChainGetBlock(ctx, otherCid)
+		return otherHeader, nil, xerrors.Errorf("chain get other block error:%s", err)
+	}
+	blockA, err := a.ChainGetBlock(ctx, otherCid)
+
+	// (c) parent-grinding fault
+	// Here extra is the "witness", a third block that shows the connection between A and B as
+	// A's sibling and B's parent.
+	// Specifically, since A is of lower height, it must be that B was mined omitting A from its tipset
+	//
+	//      B
+	//      |
+	//  [A, C]
+	if types.CidArrsEqual(blockA.Parents, blockC.Parents) && blockA.Height == blockC.Height &&
+		types.CidArrsContains(blockB.Parents, blockC.Cid()) && !types.CidArrsContains(blockB.Parents, blockA.Cid()) {
+		return blockA, blockC, xerrors.Errorf("chain get other block error:%s", err)
+	}
+
+	return nil, nil, nil
+}

documentation/en/api-v0-methods.md

@@ -4202,7 +4202,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -4233,7 +4233,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -4953,7 +4953,7 @@ Response:
     },
     "Nonce": 42,
     "Balance": "0",
-    "Address": "\u003cempty\u003e"
+    "Address": "f01234"
   }
 }
 ```
@@ -5242,7 +5242,7 @@ Response:
   },
   "Nonce": 42,
   "Balance": "0",
-  "Address": "\u003cempty\u003e"
+  "Address": "f01234"
 }
 ```
 
@@ -5430,7 +5430,6 @@ Response:
     "UpgradeRefuelHeight": 10101,
     "UpgradeTapeHeight": 10101,
     "UpgradeKumquatHeight": 10101,
-    "UpgradePriceListOopsHeight": 10101,
     "BreezeGasTampingDuration": 10101,
     "UpgradeCalicoHeight": 10101,
     "UpgradePersianHeight": 10101,
@@ -5986,6 +5985,7 @@ Response:
   "SectorSize": 34359738368,
   "WindowPoStPartitionSectors": 42,
   "ConsensusFaultElapsed": 10101,
+  "PendingOwnerAddress": "f01234",
   "Beneficiary": "f01234",
   "BeneficiaryTerm": {
     "Quota": "0",

documentation/en/api-v1-unstable-methods.md

@@ -103,6 +103,7 @@
   * [EthProtocolVersion](#EthProtocolVersion)
   * [EthSendRawTransaction](#EthSendRawTransaction)
   * [EthSubscribe](#EthSubscribe)
+  * [EthSyncing](#EthSyncing)
   * [EthUninstallFilter](#EthUninstallFilter)
   * [EthUnsubscribe](#EthUnsubscribe)
 * [Filecoin](#Filecoin)
@@ -2598,7 +2599,7 @@ Polling method for a filter, returns event logs which occurred since last poll.
 (requires write perm since timestamp of last filter execution will be written)
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -2619,7 +2620,7 @@ Returns event logs matching filter with given id.
 (requires write perm since timestamp of last filter execution will be written)
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -2990,7 +2991,7 @@ Response: `"0x0"`
 Installs a persistent filter to notify when a new block arrives.
 
 
-Perms: write
+Perms: read
 
 Inputs: `null`
 
@@ -3000,7 +3001,7 @@ Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 Installs a persistent filter based on given filter spec.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3021,7 +3022,7 @@ Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 Installs a persistent filter to notify when new messages arrive in the message pool.
 
 
-Perms: write
+Perms: read
 
 Inputs: `null`
 
@@ -3060,7 +3061,7 @@ params contains additional parameters used with the log event type
 The client will receive a stream of EthSubscriptionResponse values until EthUnsubscribe is called.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3071,11 +3072,20 @@ Inputs:
 
 Response: `"0x37690cfec6c1bf4c3b9288c7a5d783e98731e90b0a4c177c2a374c7a9427355e"`
 
+### EthSyncing
+
+
+Perms: read
+
+Inputs: `null`
+
+Response: `false`
+
 ### EthUninstallFilter
 Uninstalls a filter with given id.
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -3090,7 +3100,7 @@ Response: `true`
 Unsubscribe from a websocket subscription
 
 
-Perms: write
+Perms: read
 
 Inputs:
 ```json
@@ -5566,7 +5576,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -5597,7 +5607,7 @@ Inputs:
 Response:
 ```json
 {
-  "Channel": "\u003cempty\u003e",
+  "Channel": "f01234",
   "From": "f01234",
   "To": "f01234",
   "ConfirmedAmt": "0",
@@ -6380,7 +6390,7 @@ Response:
     },
     "Nonce": 42,
     "Balance": "0",
-    "Address": "\u003cempty\u003e"
+    "Address": "f01234"
   }
 }
 ```
@@ -6720,7 +6730,7 @@ Response:
   },
   "Nonce": 42,
   "Balance": "0",
-  "Address": "\u003cempty\u003e"
+  "Address": "f01234"
 }
 ```
 
@@ -6932,7 +6942,6 @@ Response:
     "UpgradeRefuelHeight": 10101,
     "UpgradeTapeHeight": 10101,
     "UpgradeKumquatHeight": 10101,
-    "UpgradePriceListOopsHeight": 10101,
     "BreezeGasTampingDuration": 10101,
     "UpgradeCalicoHeight": 10101,
     "UpgradePersianHeight": 10101,
@@ -7496,6 +7505,7 @@ Response:
   "SectorSize": 34359738368,
   "WindowPoStPartitionSectors": 42,
   "ConsensusFaultElapsed": 10101,
+  "PendingOwnerAddress": "f01234",
   "Beneficiary": "f01234",
   "BeneficiaryTerm": {
     "Quota": "0",

documentation/en/cli-lotus-worker.md

@@ -7,7 +7,7 @@ USAGE:
    lotus-worker [global options] command [command options] [arguments...]
 
 VERSION:
-   1.23.2
+   1.23.2-dev
 
 COMMANDS:
    run        Start lotus worker
@@ -19,12 +19,11 @@ COMMANDS:
    help, h    Shows a list of commands or help for one command
 
 GLOBAL OPTIONS:
-   --enable-gpu-proving                     enable use of GPU for mining operations (default: true) [$LOTUS_WORKER_ENABLE_GPU_PROVING]
-   --help, -h                               show help (default: false)
-   --miner-repo value, --storagerepo value  Specify miner repo path. flag storagerepo and env LOTUS_STORAGE_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
-   --version, -v                            print the version (default: false)
    --worker-repo value, --workerrepo value  Specify worker repo path. flag workerrepo and env WORKER_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusworker") [$LOTUS_WORKER_PATH, $WORKER_PATH]
-   
+   --miner-repo value, --storagerepo value  Specify miner repo path. flag storagerepo and env LOTUS_STORAGE_PATH are DEPRECATION, will REMOVE SOON (default: "~/.lotusminer") [$LOTUS_MINER_PATH, $LOTUS_STORAGE_PATH]
+   --enable-gpu-proving                     enable use of GPU for mining operations (default: true) [$LOTUS_WORKER_ENABLE_GPU_PROVING]
+   --help, -h                               show help
+   --version, -v                            print the version
 ```
 
 ## lotus-worker run
@@ -36,28 +35,29 @@ USAGE:
    lotus-worker run [command options] [arguments...]
 
 OPTIONS:
-   --addpiece                    enable addpiece (default: true) [$LOTUS_WORKER_ADDPIECE]
-   --commit                      enable commit (default: true) [$LOTUS_WORKER_COMMIT]
-   --http-server-timeout value   (default: "30s")
    --listen value                host address and port the worker api will listen on (default: "0.0.0.0:3456") [$LOTUS_WORKER_LISTEN]
-   --name value                  custom worker name (default: hostname) [$LOTUS_WORKER_NAME]
-   --no-default                  disable all default compute tasks, use the worker for storage/fetching only (default: false) [$LOTUS_WORKER_NO_DEFAULT]
    --no-local-storage            don't use storageminer repo for sector storage (default: false) [$LOTUS_WORKER_NO_LOCAL_STORAGE]
    --no-swap                     don't use swap (default: false) [$LOTUS_WORKER_NO_SWAP]
+   --name value                  custom worker name (default: hostname) [$LOTUS_WORKER_NAME]
+   --addpiece                    enable addpiece (default: true) [$LOTUS_WORKER_ADDPIECE]
+   --precommit1                  enable precommit1 (default: true) [$LOTUS_WORKER_PRECOMMIT1]
+   --unseal                      enable unsealing (default: true) [$LOTUS_WORKER_UNSEAL]
+   --precommit2                  enable precommit2 (default: true) [$LOTUS_WORKER_PRECOMMIT2]
+   --commit                      enable commit (default: true) [$LOTUS_WORKER_COMMIT]
+   --replica-update              enable replica update (default: true) [$LOTUS_WORKER_REPLICA_UPDATE]
+   --prove-replica-update2       enable prove replica update 2 (default: true) [$LOTUS_WORKER_PROVE_REPLICA_UPDATE2]
+   --regen-sector-key            enable regen sector key (default: true) [$LOTUS_WORKER_REGEN_SECTOR_KEY]
+   --sector-download             enable external sector data download (default: false) [$LOTUS_WORKER_SECTOR_DOWNLOAD]
+   --windowpost                  enable window post (default: false) [$LOTUS_WORKER_WINDOWPOST]
+   --winningpost                 enable winning post (default: false) [$LOTUS_WORKER_WINNINGPOST]
+   --no-default                  disable all default compute tasks, use the worker for storage/fetching only (default: false) [$LOTUS_WORKER_NO_DEFAULT]
    --parallel-fetch-limit value  maximum fetch operations to run in parallel (default: 5) [$LOTUS_WORKER_PARALLEL_FETCH_LIMIT]
    --post-parallel-reads value   maximum number of parallel challenge reads (0 = no limit) (default: 32) [$LOTUS_WORKER_POST_PARALLEL_READS]
    --post-read-timeout value     time limit for reading PoSt challenges (0 = no limit) (default: 0s) [$LOTUS_WORKER_POST_READ_TIMEOUT]
-   --precommit1                  enable precommit1 (default: true) [$LOTUS_WORKER_PRECOMMIT1]
-   --precommit2                  enable precommit2 (default: true) [$LOTUS_WORKER_PRECOMMIT2]
-   --prove-replica-update2       enable prove replica update 2 (default: true) [$LOTUS_WORKER_PROVE_REPLICA_UPDATE2]
-   --regen-sector-key            enable regen sector key (default: true) [$LOTUS_WORKER_REGEN_SECTOR_KEY]
-   --replica-update              enable replica update (default: true) [$LOTUS_WORKER_REPLICA_UPDATE]
-   --sector-download             enable external sector data download (default: false) [$LOTUS_WORKER_SECTOR_DOWNLOAD]
    --timeout value               used when 'listen' is unspecified. must be a valid duration recognized by golang's time.ParseDuration function (default: "30m") [$LOTUS_WORKER_TIMEOUT]
-   --unseal                      enable unsealing (default: true) [$LOTUS_WORKER_UNSEAL]
-   --windowpost                  enable window post (default: false) [$LOTUS_WORKER_WINDOWPOST]
-   --winningpost                 enable winning post (default: false) [$LOTUS_WORKER_WINNINGPOST]
-   
+   --http-server-timeout value   (default: "30s")
+   --data-cid                    Run the data-cid task. true|false (default: inherits --addpiece)
+   --help, -h                    show help
 ```
 
 ## lotus-worker stop
@@ -69,8 +69,7 @@ USAGE:
    lotus-worker stop [command options] [arguments...]
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ## lotus-worker info
@@ -82,8 +81,7 @@ USAGE:
    lotus-worker info [command options] [arguments...]
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ## lotus-worker storage
@@ -95,14 +93,13 @@ USAGE:
    lotus-worker storage command [command options] [arguments...]
 
 COMMANDS:
-     attach     attach local storage path
-     detach     detach local storage path
-     redeclare  redeclare sectors in a local storage path
-     help, h    Shows a list of commands or help for one command
+   attach     attach local storage path
+   detach     detach local storage path
+   redeclare  redeclare sectors in a local storage path
+   help, h    Shows a list of commands or help for one command
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ### lotus-worker storage attach
@@ -114,14 +111,14 @@ USAGE:
    lotus-worker storage attach [command options] [arguments...]
 
 OPTIONS:
-   --allow-to value [ --allow-to value ]  path groups allowed to pull data from this path (allow all if not specified)
-   --groups value [ --groups value ]      path group names
    --init                                 initialize the path first (default: false)
-   --max-storage value                    (for init) limit storage space for sectors (expensive for very large paths!)
+   --weight value                         (for init) path weight (default: 10)
    --seal                                 (for init) use path for sealing (default: false)
    --store                                (for init) use path for long-term storage (default: false)
-   --weight value                         (for init) path weight (default: 10)
-   
+   --max-storage value                    (for init) limit storage space for sectors (expensive for very large paths!)
+   --groups value [ --groups value ]      path group names
+   --allow-to value [ --allow-to value ]  path groups allowed to pull data from this path (allow all if not specified)
+   --help, -h                             show help
 ```
 
 ### lotus-worker storage detach
@@ -134,7 +131,7 @@ USAGE:
 
 OPTIONS:
    --really-do-it  (default: false)
-   
+   --help, -h      show help
 ```
 
 ### lotus-worker storage redeclare
@@ -146,10 +143,10 @@ USAGE:
    lotus-worker storage redeclare [command options] [arguments...]
 
 OPTIONS:
-   --all           redeclare all storage paths (default: false)
-   --drop-missing  Drop index entries with missing files (default: false)
    --id value      storage path ID
-   
+   --all           redeclare all storage paths (default: false)
+   --drop-missing  Drop index entries with missing files (default: true)
+   --help, -h      show help
 ```
 
 ## lotus-worker resources
@@ -161,9 +158,9 @@ USAGE:
    lotus-worker resources [command options] [arguments...]
 
 OPTIONS:
-   --all      print all resource envvars (default: false)
-   --default  print default resource envvars (default: false)
-   
+   --all       print all resource envvars (default: false)
+   --default   print default resource envvars (default: false)
+   --help, -h  show help
 ```
 
 ## lotus-worker tasks
@@ -175,13 +172,12 @@ USAGE:
    lotus-worker tasks command [command options] [arguments...]
 
 COMMANDS:
-     enable   Enable a task type
-     disable  Disable a task type
-     help, h  Shows a list of commands or help for one command
+   enable   Enable a task type
+   disable  Disable a task type
+   help, h  Shows a list of commands or help for one command
 
 OPTIONS:
-   --help, -h  show help (default: false)
-   
+   --help, -h  show help
 ```
 
 ### lotus-worker tasks enable
@@ -193,8 +189,8 @@ USAGE:
    lotus-worker tasks enable [command options] --all | [UNS|C2|PC2|PC1|PR2|RU|AP|DC|GSK]
 
 OPTIONS:
-   --all  Enable all task types (default: false)
-   
+   --all       Enable all task types (default: false)
+   --help, -h  show help
 ```
 
 ### lotus-worker tasks disable
@@ -206,6 +202,6 @@ USAGE:
    lotus-worker tasks disable [command options] --all | [UNS|C2|PC2|PC1|PR2|RU|AP|DC|GSK]
 
 OPTIONS:
-   --all  Disable all task types (default: false)
-   
+   --all       Disable all task types (default: false)
+   --help, -h  show help
 ```

documentation/en/default-lotus-config.toml

@@ -38,7 +38,7 @@
   #
   # type: []string
   # env var: LOTUS_LIBP2P_LISTENADDRESSES
-  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0"]
+  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0", "/ip4/0.0.0.0/udp/0/quic-v1", "/ip6/::/udp/0/quic-v1", "/ip4/0.0.0.0/udp/0/quic-v1/webtransport", "/ip6/::/udp/0/quic-v1/webtransport"]
 
   # Addresses to explicitally announce to other peers. If not specified,
   # all interface addresses are announced

documentation/en/default-lotus-miner-config.toml

@@ -38,7 +38,7 @@
   #
   # type: []string
   # env var: LOTUS_LIBP2P_LISTENADDRESSES
-  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0"]
+  #ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0", "/ip4/0.0.0.0/udp/0/quic-v1", "/ip6/::/udp/0/quic-v1", "/ip4/0.0.0.0/udp/0/quic-v1/webtransport", "/ip6/::/udp/0/quic-v1/webtransport"]
 
   # Addresses to explicitally announce to other peers. If not specified,
   # all interface addresses are announced
@@ -515,15 +515,7 @@
   # env var: LOTUS_SEALING_MINUPGRADESECTOREXPIRATION
   #MinUpgradeSectorExpiration = 0
 
-  # When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-  # be selected based on lowest initial pledge.
-  # 
-  # Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-  # selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-  # where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-  # 
-  # Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-  # initial pledge - upgrade sectors will always be chosen based on longest expiration
+  # DEPRECATED: Target expiration is no longer used
   #
   # type: uint64
   # env var: LOTUS_SEALING_MINTARGETUPGRADESECTOREXPIRATION

documentation/misc/gas_balancing.md

@@ -2,8 +2,8 @@
 
 The gas balancing process targets to set gas costs of syscalls to be in line with
 10 gas per nanosecond on reference hardware.
-The process can be either performed for all syscalls based on existing messages and chain or targeted
-at single syscall.
+The process can be either performed for all syscalls based on existing messages and chains or targeted
+at a single syscall.
 
 #### Reference hardware
 
@@ -12,14 +12,14 @@ may be subject to change.
 
 ### Complete gas balancing
 
-Complete gas balancing is performed using `lotus-bench` the process is based on importing a chain export
+Complete gas balancing is performed using a `lotus-bench` the process is based on importing a chain export
 and collecting gas traces which are later aggregated.
 
 Before building `lotus-bench` make sure `EnableDetailedTracing` in `chain/vm/runtime.go` is set to `true`.
 
 The process can be started using `./lotus-bench import` with `--car` flag set to the location of
-CAR chain export. `--start-epoch` and `--end-epoch` can be used to to limit the range of epochs to run
-the benchmark. Note that state tree of `start-epoch` needs to be in the CAR file or has to be previously computed
+CAR chain export. `--start-epoch` and `--end-epoch` can be used to limit the range of epochs to run
+the benchmark. Note that the state tree of `start-epoch` needs to be in the CAR file or has to be previously computed
 to work.
 
 The output will be a `bench.json` file containing information about every syscall invoked
@@ -29,7 +29,7 @@ spare space.
 After the bench run is complete the `bench.json` file can be analyzed with `./lotus-bench import analyze bench.json`.
 
 It will compute means, standard deviations and co-variances (when applicable) of syscall runtimes. 
-The output is in nanoseconds, so the gas values for syscalls should be 10x that. In cases where co-variance of
+The output is in nanoseconds, so the gas values for syscalls should be 10x that. In cases where the co-variance of
 execution time to some parameter is evaluated, the strength of the correlation should be taken into account.
 
 #### Special cases
@@ -40,15 +40,15 @@ during block execution (when gas traces are formed) objects are only written to
 
 ### Targeted gas balancing
 
-In some cases complete gas balancing is infeasible, either new syscall gets introduced or
+In some cases complete gas balancing is infeasible, either a new syscall gets introduced or
 complete balancing is too time consuming.
 
-In these cases the recommended way to estimate gas for given syscall is to perform an `in-vivo` benchmark.
+In these cases, the recommended way to estimate gas for a given syscall is to perform an `in-vivo` benchmark.
 In the past `in-vitro` as in standalone benchmarks were found to be highly inaccurate when compared to results
 of real execution.
 
-A in-vivo benchmark can be performed by running an example of such syscall during block execution.
-The best place to hook-in such benchmark is message execution loop in
-`chain/stmgr/stmgr.go` in `ApplyBlocks()`. Depending of time required to complete the syscall it might be
+An in-vivo benchmark can be performed by running an example of such a syscall during block execution.
+The best place to hook-in such a benchmark is the message execution loop in
+`chain/stmgr/stmgr.go` in `ApplyBlocks()`. Depending on the time required to complete the syscall it might be
 advisable to run the execution only once every few messages.
 

gateway/node.go

@@ -43,6 +43,9 @@ const (
 // TargetAPI defines the API methods that the Node depends on
 // (to make it easy to mock for tests)
 type TargetAPI interface {
+	MpoolPending(context.Context, types.TipSetKey) ([]*types.SignedMessage, error)
+	ChainGetBlock(context.Context, cid.Cid) (*types.BlockHeader, error)
+	MinerGetBaseInfo(context.Context, address.Address, abi.ChainEpoch, types.TipSetKey) (*api.MiningBaseInfo, error)
 	GasEstimateGasPremium(context.Context, uint64, address.Address, int64, types.TipSetKey) (types.BigInt, error)
 	StateReplay(context.Context, types.TipSetKey, cid.Cid) (*api.InvocResult, error)
 	StateMinerSectorCount(context.Context, address.Address, types.TipSetKey) (api.MinerSectors, error)
@@ -115,6 +118,7 @@ type TargetAPI interface {
 	EthGetStorageAt(ctx context.Context, address ethtypes.EthAddress, position ethtypes.EthBytes, blkParam string) (ethtypes.EthBytes, error)
 	EthGetBalance(ctx context.Context, address ethtypes.EthAddress, blkParam string) (ethtypes.EthBigInt, error)
 	EthChainId(ctx context.Context) (ethtypes.EthUint64, error)
+	EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error)
 	NetVersion(ctx context.Context) (string, error)
 	NetListening(ctx context.Context) (bool, error)
 	EthProtocolVersion(ctx context.Context) (ethtypes.EthUint64, error)

gateway/proxy_eth.go

@@ -252,6 +252,14 @@ func (gw *Node) EthChainId(ctx context.Context) (ethtypes.EthUint64, error) {
 	return gw.target.EthChainId(ctx)
 }
 
+func (gw *Node) EthSyncing(ctx context.Context) (ethtypes.EthSyncingResult, error) {
+	if err := gw.limit(ctx, basicRateLimitTokens); err != nil {
+		return ethtypes.EthSyncingResult{}, err
+	}
+
+	return gw.target.EthSyncing(ctx)
+}
+
 func (gw *Node) NetVersion(ctx context.Context) (string, error) {
 	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
 		return "", err

gateway/proxy_fil.go

@@ -23,6 +23,33 @@ import (
 	"github.com/filecoin-project/lotus/node/modules/dtypes"
 )
 
+func (gw *Node) MpoolPending(ctx context.Context, tsk types.TipSetKey) ([]*types.SignedMessage, error) {
+	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
+		return nil, err
+	}
+	if err := gw.checkTipsetKey(ctx, tsk); err != nil {
+		return nil, err
+	}
+	return gw.target.MpoolPending(ctx, tsk)
+}
+
+func (gw *Node) ChainGetBlock(ctx context.Context, c cid.Cid) (*types.BlockHeader, error) {
+	if err := gw.limit(ctx, chainRateLimitTokens); err != nil {
+		return nil, err
+	}
+	return gw.target.ChainGetBlock(ctx, c)
+}
+
+func (gw *Node) MinerGetBaseInfo(ctx context.Context, addr address.Address, h abi.ChainEpoch, tsk types.TipSetKey) (*api.MiningBaseInfo, error) {
+	if err := gw.limit(ctx, stateRateLimitTokens); err != nil {
+		return nil, err
+	}
+	if err := gw.checkTipsetKey(ctx, tsk); err != nil {
+		return nil, err
+	}
+	return gw.target.MinerGetBaseInfo(ctx, addr, h, tsk)
+}
+
 func (gw *Node) StateReplay(ctx context.Context, tsk types.TipSetKey, c cid.Cid) (*api.InvocResult, error) {
 	if err := gw.limit(ctx, chainRateLimitTokens); err != nil {
 		return nil, err

go.mod

@@ -38,11 +38,11 @@ require (
 	github.com/filecoin-project/go-cbor-util v0.0.1
 	github.com/filecoin-project/go-commp-utils v0.1.3
 	github.com/filecoin-project/go-crypto v0.0.1
-	github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4
+	github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7
 	github.com/filecoin-project/go-fil-commcid v0.1.0
 	github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
-	github.com/filecoin-project/go-fil-markets v1.27.0-rc1
-	github.com/filecoin-project/go-jsonrpc v0.2.3
+	github.com/filecoin-project/go-fil-markets v1.28.2
+	github.com/filecoin-project/go-jsonrpc v0.3.1
 	github.com/filecoin-project/go-padreader v0.0.1
 	github.com/filecoin-project/go-paramfetch v0.0.4
 	github.com/filecoin-project/go-state-types v0.11.1
@@ -87,7 +87,7 @@ require (
 	github.com/ipfs/go-ds-leveldb v0.5.0
 	github.com/ipfs/go-ds-measure v0.2.0
 	github.com/ipfs/go-fs-lock v0.0.7
-	github.com/ipfs/go-graphsync v0.14.3
+	github.com/ipfs/go-graphsync v0.14.6
 	github.com/ipfs/go-ipfs-blockstore v1.3.0
 	github.com/ipfs/go-ipfs-blocksutil v0.0.1
 	github.com/ipfs/go-ipfs-chunker v0.0.5
@@ -111,8 +111,8 @@ require (
 	github.com/ipld/go-codec-dagpb v1.6.0
 	github.com/ipld/go-ipld-prime v0.20.0
 	github.com/ipld/go-ipld-selector-text-lite v0.0.1
-	github.com/ipni/index-provider v0.11.0
-	github.com/ipni/storetheindex v0.5.10
+	github.com/ipni/go-libipni v0.0.8
+	github.com/ipni/index-provider v0.12.0
 	github.com/kelseyhightower/envconfig v1.4.0
 	github.com/koalacxr/quantile v0.0.1
 	github.com/libp2p/go-buffer-pool v0.1.0
@@ -144,7 +144,7 @@ require (
 	github.com/raulk/go-watchdog v1.3.0
 	github.com/stretchr/testify v1.8.2
 	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
-	github.com/urfave/cli/v2 v2.16.3
+	github.com/urfave/cli/v2 v2.25.5
 	github.com/whyrusleeping/bencher v0.0.0-20190829221104-bb6607aa8bba
 	github.com/whyrusleeping/cbor-gen v0.0.0-20230126041949-52956bd4c9aa
 	github.com/whyrusleeping/ledger-filecoin-go v0.9.1-0.20201010031517-c3dcc1bddce4
@@ -286,7 +286,7 @@ require (
 	github.com/mr-tron/base58 v1.2.0 // indirect
 	github.com/multiformats/go-base36 v0.2.0 // indirect
 	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
-	github.com/multiformats/go-multicodec v0.8.1 // indirect
+	github.com/multiformats/go-multicodec v0.9.0 // indirect
 	github.com/multiformats/go-multistream v0.4.1 // indirect
 	github.com/nikkolasg/hexjson v0.1.0 // indirect
 	github.com/nkovacs/streamquote v1.0.0 // indirect

go.sum

@@ -312,8 +312,8 @@ github.com/filecoin-project/go-commp-utils/nonffi v0.0.0-20220905160352-62059082
 github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
 github.com/filecoin-project/go-crypto v0.0.1 h1:AcvpSGGCgjaY8y1az6AMfKQWreF/pWO2JJGLl6gCq6o=
 github.com/filecoin-project/go-crypto v0.0.1/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
-github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4 h1:Y5RMvFT4OthsAhDx7xKfkJ5QdiWq9Ox7N46Mi0PF0PE=
-github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc4/go.mod h1:1WDoUgWYB2KvogfPTdDmoyBTa9cb9+oGwqKCCipnJeY=
+github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7 h1:v+zJS5B6pA3ptWZS4t8tbt1Hz9qENnN4nVr1w99aSWc=
+github.com/filecoin-project/go-data-transfer/v2 v2.0.0-rc7/go.mod h1:V3Y4KbttaCwyg1gwkP7iai8CbQx4mZUGjd3h9GZWLKE=
 github.com/filecoin-project/go-ds-versioning v0.1.2 h1:to4pTadv3IeV1wvgbCbN6Vqd+fu+7tveXgv/rCEZy6w=
 github.com/filecoin-project/go-ds-versioning v0.1.2/go.mod h1:C9/l9PnB1+mwPa26BBVpCjG/XQCB0yj/q5CK2J8X1I4=
 github.com/filecoin-project/go-fil-commcid v0.0.0-20201016201715-d41df56b4f6a/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
@@ -321,8 +321,8 @@ github.com/filecoin-project/go-fil-commcid v0.1.0 h1:3R4ds1A9r6cr8mvZBfMYxTS88Oq
 github.com/filecoin-project/go-fil-commcid v0.1.0/go.mod h1:Eaox7Hvus1JgPrL5+M3+h7aSPHc0cVqpSxA+TxIEpZQ=
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0 h1:imrrpZWEHRnNqqv0tN7LXep5bFEVOVmQWHJvl2mgsGo=
 github.com/filecoin-project/go-fil-commp-hashhash v0.1.0/go.mod h1:73S8WSEWh9vr0fDJVnKADhfIv/d6dCbAGaAGWbdJEI8=
-github.com/filecoin-project/go-fil-markets v1.27.0-rc1 h1:SYXKFONg6IaQlmXEqAEaPtb/xX57dZYMkx3LwjsLr7w=
-github.com/filecoin-project/go-fil-markets v1.27.0-rc1/go.mod h1:JkW4rU0+LqfO/DLi/wyR58AqxaUdnlSofWeu8un2XLU=
+github.com/filecoin-project/go-fil-markets v1.28.2 h1:Ev9o8BYow+lo97Bwc6oOmZ2OxdiHeIDCQsfF/w/Vldc=
+github.com/filecoin-project/go-fil-markets v1.28.2/go.mod h1:qy9LNu9t77I184VB6Pa4WKRtGfB8Vl0t8zfOLHkDqWY=
 github.com/filecoin-project/go-hamt-ipld v0.1.5 h1:uoXrKbCQZ49OHpsTCkrThPNelC4W3LPEk0OrS/ytIBM=
 github.com/filecoin-project/go-hamt-ipld v0.1.5/go.mod h1:6Is+ONR5Cd5R6XZoCse1CWaXZc0Hdb/JeX+EQCQzX24=
 github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0 h1:b3UDemBYN2HNfk3KOXNuxgTTxlWi3xVvbQP0IT38fvM=
@@ -330,8 +330,8 @@ github.com/filecoin-project/go-hamt-ipld/v2 v2.0.0/go.mod h1:7aWZdaQ1b16BVoQUYR+
 github.com/filecoin-project/go-hamt-ipld/v3 v3.0.1/go.mod h1:gXpNmr3oQx8l3o7qkGyDjJjYSRX7hp/FGOStdqrWyDI=
 github.com/filecoin-project/go-hamt-ipld/v3 v3.1.0 h1:rVVNq0x6RGQIzCo1iiJlGFm9AGIZzeifggxtKMU7zmI=
 github.com/filecoin-project/go-hamt-ipld/v3 v3.1.0/go.mod h1:bxmzgT8tmeVQA1/gvBwFmYdT8SOFUwB3ovSUfG1Ux0g=
-github.com/filecoin-project/go-jsonrpc v0.2.3 h1:xdixRQfLbD8gt56dpBNKiya9gvI/79nNM13IHKTfm5E=
-github.com/filecoin-project/go-jsonrpc v0.2.3/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM=
+github.com/filecoin-project/go-jsonrpc v0.3.1 h1:qwvAUc5VwAkooquKJmfz9R2+F8znhiqcNHYjEp/NM10=
+github.com/filecoin-project/go-jsonrpc v0.3.1/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM=
 github.com/filecoin-project/go-padreader v0.0.0-20200903213702-ed5fae088b20/go.mod h1:mPn+LRRd5gEKNAtc+r3ScpW2JRU/pj4NBKdADYWHiak=
 github.com/filecoin-project/go-padreader v0.0.1 h1:8h2tVy5HpoNbr2gBRr+WD6zV6VD6XHig+ynSGJg8ZOs=
 github.com/filecoin-project/go-padreader v0.0.1/go.mod h1:VYVPJqwpsfmtoHnAmPx6MUwmrK6HIcDqZJiuZhtmfLQ=
@@ -738,8 +738,8 @@ github.com/ipfs/go-filestore v1.2.0/go.mod h1:HLJrCxRXquTeEEpde4lTLMaE/MYJZD7WHL
 github.com/ipfs/go-fs-lock v0.0.6/go.mod h1:OTR+Rj9sHiRubJh3dRhD15Juhd/+w6VPOY28L7zESmM=
 github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U=
 github.com/ipfs/go-fs-lock v0.0.7/go.mod h1:Js8ka+FNYmgQRLrRXzU3CB/+Csr1BwrRilEcvYrHhhc=
-github.com/ipfs/go-graphsync v0.14.3 h1:IXH9S7AraMQ0J6Fzcl8rqSPqLn+es33bD8OW2KNyU/o=
-github.com/ipfs/go-graphsync v0.14.3/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1taIQX/aHbBQo=
+github.com/ipfs/go-graphsync v0.14.6 h1:NPxvuUy4Z08Mg8dwpBzwgbv/PGLIufSJ1sle6iAX8yo=
+github.com/ipfs/go-graphsync v0.14.6/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1taIQX/aHbBQo=
 github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08=
 github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw=
 github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE=
@@ -885,10 +885,10 @@ github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20211210234204-ce2a1c70cd
 github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20230102063945-1a409dc236dd h1:gMlw/MhNr2Wtp5RwGdsW23cs+yCuj9k2ON7i9MiJlRo=
 github.com/ipld/go-ipld-selector-text-lite v0.0.1 h1:lNqFsQpBHc3p5xHob2KvEg/iM5dIFn6iw4L/Hh+kS1Y=
 github.com/ipld/go-ipld-selector-text-lite v0.0.1/go.mod h1:U2CQmFb+uWzfIEF3I1arrDa5rwtj00PrpiwwCO+k1RM=
-github.com/ipni/index-provider v0.11.0 h1:q2PdK6JpYB9bzlntfkRYNBjhg4Qtko5+iXRonO88TAg=
-github.com/ipni/index-provider v0.11.0/go.mod h1:gB/wN4Mdz4MzikQubjyRRV97iS5BkD4FKB0U/bF/dY4=
-github.com/ipni/storetheindex v0.5.10 h1:r97jIZsXPuwQvePJQuStu2a/kn+Zn8X4MAdA0rU2Pu4=
-github.com/ipni/storetheindex v0.5.10/go.mod h1:SJKFCnSx4X/4ekQuZvq8pVU/7tmxkEv632Qmgu3m2bQ=
+github.com/ipni/go-libipni v0.0.8 h1:0wLfZRSBG84swmZwmaLKul/iB/FlBkkl9ZcR1ub+Z+w=
+github.com/ipni/go-libipni v0.0.8/go.mod h1:paYP9U4N3/vOzGCuN9kU972vtvw9JUcQjOKyiCFGwRk=
+github.com/ipni/index-provider v0.12.0 h1:R3F6dxxKNv4XkE4GJZNLOG0bDEbBQ/S5iztXwSD8jhQ=
+github.com/ipni/index-provider v0.12.0/go.mod h1:GhyrADJp7n06fqoc1djzkvL4buZYHzV8SoWrlxEo5F4=
 github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52 h1:QG4CGBqCeuBo6aZlGAamSkxWdgWfZGeE49eUOWJPA4c=
 github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52/go.mod h1:fdg+/X9Gg4AsAIzWpEHwnqd+QY3b7lajxyjE1m4hkq4=
 github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA=
@@ -1325,8 +1325,8 @@ github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6o
 github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ=
 github.com/multiformats/go-multicodec v0.3.1-0.20210902112759-1539a079fd61/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ=
 github.com/multiformats/go-multicodec v0.6.0/go.mod h1:GUC8upxSBE4oG+q3kWZRw/+6yC1BqO550bjhWsJbZlw=
-github.com/multiformats/go-multicodec v0.8.1 h1:ycepHwavHafh3grIbR1jIXnKCsFm0fqsfEOsJ8NtKE8=
-github.com/multiformats/go-multicodec v0.8.1/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
+github.com/multiformats/go-multicodec v0.9.0 h1:pb/dlPnzee/Sxv/j4PmkDRxCOi3hXTz3IbPKOXWJkmg=
+github.com/multiformats/go-multicodec v0.9.0/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
 github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
 github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po=
 github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
@@ -1630,8 +1630,8 @@ github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
 github.com/urfave/cli v1.22.2/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
 github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
 github.com/urfave/cli/v2 v2.0.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
-github.com/urfave/cli/v2 v2.16.3 h1:gHoFIwpPjoyIMbJp/VFd+/vuD0dAgFK4B6DpEMFJfQk=
-github.com/urfave/cli/v2 v2.16.3/go.mod h1:1CNUng3PtjQMtRzJO4FMXBQvkGtuYRxxiR9xMa7jMwI=
+github.com/urfave/cli/v2 v2.25.5 h1:d0NIAyhh5shGscroL7ek/Ya9QYQE0KNabJgiUinIQkc=
+github.com/urfave/cli/v2 v2.25.5/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasttemplate v1.0.1 h1:tY9CJiPnMXf1ERmG2EyK7gNUd+c6RKGD0IfU8WdUSz8=

itests/deals_partial_retrieval_dm-level_test.go

@@ -48,9 +48,9 @@ func TestDMLevelPartialRetrieval(t *testing.T) {
 	ctx := context.Background()
 
 	kit.QuietMiningLogs()
-	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC(), kit.MockProofs())
+	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC())
 	dh := kit.NewDealHarness(t, client, miner, miner)
-	ens.InterconnectAll().BeginMining(50 * time.Millisecond)
+	ens.InterconnectAll().BeginMiningMustPost(50 * time.Millisecond)
 
 	_, err := client.ClientImport(ctx, api.FileRef{Path: adlFixtureCar, IsCAR: true})
 	require.NoError(t, err)

itests/eth_balance_test.go

@@ -121,15 +121,24 @@ func TestEthBalanceCorrectLookup(t *testing.T) {
 	require.NoError(t, err)
 	require.True(t, ml.Receipt.ExitCode.IsSuccess())
 
-	bal, err := client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height-2), 10))
+	execTs, err := client.ChainGetTipSet(ctx, ml.TipSet)
+	require.NoError(t, err)
+
+	inclTs, err := client.ChainGetTipSet(ctx, execTs.Parents())
+	require.NoError(t, err)
+
+	inclTsParents, err := client.ChainGetTipSet(ctx, inclTs.Parents())
+	require.NoError(t, err)
+
+	bal, err := client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(inclTsParents.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, int64(0), bal.Int64())
 
-	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height-1), 10))
+	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(inclTs.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, val, bal.Int64())
 
-	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(ml.Height), 10))
+	bal, err = client.EVM().EthGetBalance(ctx, ethAddr, strconv.FormatInt(int64(execTs.Height()), 10))
 	require.NoError(t, err)
 	require.Equal(t, val, bal.Int64())
 }

itests/eth_deploy_test.go

@@ -29,18 +29,13 @@ import (
 // TestDeployment smoke tests the deployment of a contract via the
 // Ethereum JSON-RPC endpoint, from an EEOA.
 func TestDeployment(t *testing.T) {
-	// TODO::FVM @raulk the contract installation and invocation can be lifted into utility methods
-	// He who writes the second test, shall do that.
-	// kit.QuietMiningLogs()
-
-	// reasonable blocktime so that the tx sits in the mpool for a bit during the test.
-	// although this is non-deterministic...
-	blockTime := 1 * time.Second
+	blockTime := 100 * time.Millisecond
 	client, _, ens := kit.EnsembleMinimal(
 		t,
 		kit.MockProofs(),
 		kit.ThroughRPC())
-	ens.InterconnectAll().BeginMining(blockTime)
+
+	miners := ens.InterconnectAll().BeginMining(blockTime)
 
 	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
 	defer cancel()
@@ -93,6 +88,11 @@ func TestDeployment(t *testing.T) {
 	pendingFilter, err := client.EthNewPendingTransactionFilter(ctx)
 	require.NoError(t, err)
 
+	// Pause so we can test that everything works while the message is in the message pool.
+	for _, miner := range miners {
+		miner.Pause()
+	}
+
 	hash := client.EVM().SubmitTransaction(ctx, &tx)
 
 	mpoolTx, err := client.EthGetTransactionByHash(ctx, &hash)
@@ -103,31 +103,40 @@ func TestDeployment(t *testing.T) {
 	require.Equal(t, hash, mpoolTx.Hash)
 
 	// these fields should be nil because the tx hasn't landed on chain.
-	// TODO::FVM @raulk We can either skip the assertion if the msg has already
-	//  landed, or pause mining between the embryo creation and this assertion.
 	require.Nil(t, mpoolTx.BlockNumber)
 	require.Nil(t, mpoolTx.BlockHash)
 	require.Nil(t, mpoolTx.TransactionIndex)
 
+	// We should be able to get the message CID immediately.
+	mCid, err := client.EthGetMessageCidByTransactionHash(ctx, &hash)
+	require.NoError(t, err)
+	require.NotNil(t, mCid)
+
+	// ... and it should map back to the transaction hash.
+	mHash, err := client.EthGetTransactionHashByCid(ctx, *mCid)
+	require.NoError(t, err)
+	require.NotNil(t, mHash)
+	require.Equal(t, hash, *mHash)
+
 	changes, err := client.EthGetFilterChanges(ctx, pendingFilter)
 	require.NoError(t, err)
 	require.Len(t, changes.Results, 1)
 	require.Equal(t, hash.String(), changes.Results[0])
 
-	var receipt *api.EthTxReceipt
-	for i := 0; i < 20; i++ {
-		// TODO::FVM @raulk The right time to exit this loop isn't after
-		//  20 iterations, but when StateWaitMsg returns -- let's wait on that
-		//  event while continuing to make this assertion
-		receipt, err = client.EthGetTransactionReceipt(ctx, hash)
-		if err != nil || receipt == nil {
-			time.Sleep(500 * time.Millisecond)
-			continue
-		}
-		break
+	// Unpause mining.
+	for _, miner := range miners {
+		miner.Restart()
 	}
+
+	// Wait for the message to land.
+	_, err = client.StateWaitMsg(ctx, *mCid, 3, api.LookbackNoLimit, false)
+	require.NoError(t, err)
+
+	// Then lookup the receipt.
+	receipt, err := client.EthGetTransactionReceipt(ctx, hash)
 	require.NoError(t, err)
 	require.NotNil(t, receipt)
+
 	// logs must be an empty array, not a nil value, to avoid tooling compatibility issues
 	require.Empty(t, receipt.Logs)
 	// a correctly formed logs bloom, albeit empty, has 256 zeroes
@@ -173,21 +182,16 @@ func TestDeployment(t *testing.T) {
 	require.Nil(t, err)
 	require.True(t, reflect.DeepEqual(block1, block2))
 
-	// should be able to get the block using latest as well
-	block3, err := client.EthGetBlockByNumber(ctx, "latest", false)
-	require.Nil(t, err)
-	require.True(t, reflect.DeepEqual(block2, block3))
-
 	// verify that the block contains full tx objects
-	block4, err := client.EthGetBlockByHash(ctx, *chainTx.BlockHash, true)
+	block3, err := client.EthGetBlockByHash(ctx, *chainTx.BlockHash, true)
 	require.Nil(t, err)
-	require.Equal(t, block4.Hash, *chainTx.BlockHash)
-	require.Equal(t, block4.Number, *chainTx.BlockNumber)
+	require.Equal(t, block3.Hash, *chainTx.BlockHash)
+	require.Equal(t, block3.Number, *chainTx.BlockNumber)
 
 	// the call went through json-rpc and the response was unmarshaled
 	// into map[string]interface{}, so it has to be converted into ethtypes.EthTx
 	var foundTx *ethtypes.EthTx
-	for _, obj := range block4.Transactions {
+	for _, obj := range block3.Transactions {
 		j, err := json.Marshal(obj)
 		require.Nil(t, err)
 
@@ -202,10 +206,10 @@ func TestDeployment(t *testing.T) {
 	require.NotNil(t, foundTx)
 	require.True(t, reflect.DeepEqual(*foundTx, *chainTx))
 
-	// make sure the block got from EthGetBlockByNumber is the same
-	block5, err := client.EthGetBlockByNumber(ctx, blkNum, true)
+	// make sure the _full_ block got from EthGetBlockByNumber is the same
+	block4, err := client.EthGetBlockByNumber(ctx, blkNum, true)
 	require.Nil(t, err)
-	require.True(t, reflect.DeepEqual(block4, block5))
+	require.True(t, reflect.DeepEqual(block3, block4))
 
 	// Verify that the deployer is now an account.
 	client.AssertActorType(ctx, deployer, manifest.EthAccountKey)

itests/eth_fee_history_test.go

@@ -50,19 +50,6 @@ func TestEthFeeHistory(t *testing.T) {
 	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
 	defer cancel()
 
-	heads, err := client.ChainNotify(ctx)
-	require.NoError(err)
-
-	// Save the full view of the tipsets to calculate the answer when there are null rounds
-	tsHeights := []int{1}
-	go func() {
-		for chg := range heads {
-			for _, c := range chg {
-				tsHeights = append(tsHeights, int(c.Val.Height()))
-			}
-		}
-	}()
-
 	miner := ens.InterconnectAll().BeginMining(blockTime)
 
 	client.WaitTillChain(ctx, kit.HeightAtLeast(7))
@@ -89,6 +76,16 @@ func TestEthFeeHistory(t *testing.T) {
 		}
 	}()
 
+	currTs, err := client.ChainHead(ctx)
+	require.NoError(err)
+
+	var tsHeights []int
+	for currTs.Height() != 0 {
+		tsHeights = append(tsHeights, int(currTs.Height()))
+		currTs, err = client.ChainGetTipSet(ctx, currTs.Parents())
+		require.NoError(err)
+	}
+
 	sort.Ints(tsHeights)
 
 	// because of the deferred execution, the last tipset is not executed yet,

itests/eth_filter_test.go

@@ -425,6 +425,11 @@ func TestEthNewFilterDefaultSpec(t *testing.T) {
 	elogs, err := parseEthLogsFromFilterResult(res)
 	require.NoError(err)
 	AssertEthLogs(t, elogs, expected, received)
+
+	// giving a number to fromBlock/toBlock needs to be in hex format, so make sure passing in decimal fails
+	blockNrInDecimal := "2812200"
+	_, err = client.EthNewFilter(ctx, &ethtypes.EthFilterSpec{FromBlock: &blockNrInDecimal, ToBlock: &blockNrInDecimal})
+	require.Error(err, "expected error when fromBlock/toBlock is not in hex format")
 }
 
 func TestEthGetLogsBasic(t *testing.T) {

itests/kit/blockminer.go

@@ -184,7 +184,8 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 
 			dlinfo, err := bm.miner.FullNode.StateMinerProvingDeadline(ctx, bm.miner.ActorAddr, ts.Key())
 			require.NoError(bm.t, err)
-			if ts.Height()+1+abi.ChainEpoch(nulls) >= dlinfo.Last() { // Next block brings us past the last epoch in dline, we need to wait for miner to post
+			if ts.Height()+5+abi.ChainEpoch(nulls) >= dlinfo.Last() { // Next block brings us past the last epoch in dline, we need to wait for miner to post
+				bm.t.Logf("forcing post to get in before deadline closes at %d", dlinfo.Last())
 				bm.forcePoSt(ctx, ts, dlinfo)
 			}
 
@@ -216,7 +217,8 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 				}
 				if !success {
 					// if we are mining a new null block and it brings us past deadline boundary we need to wait for miner to post
-					if ts.Height()+1+abi.ChainEpoch(nulls+i) >= dlinfo.Last() {
+					if ts.Height()+5+abi.ChainEpoch(nulls+i) >= dlinfo.Last() {
+						bm.t.Logf("forcing post to get in before deadline closes at %d", dlinfo.Last())
 						bm.forcePoSt(ctx, ts, dlinfo)
 					}
 				}
@@ -236,7 +238,7 @@ func (bm *BlockMiner) MineBlocksMustPost(ctx context.Context, blocktime time.Dur
 					break
 				}
 
-				require.NotEqual(bm.t, i, nloops-1, "block never managed to sync to node")
+				require.NotEqual(bm.t, i, nloops-1, "block at height %d never managed to sync to node, which is at height %d", target, ts.Height())
 				time.Sleep(time.Millisecond * 10)
 			}
 
@@ -348,7 +350,7 @@ func (bm *BlockMiner) MineUntilBlock(ctx context.Context, fn *TestFullNode, cb f
 					break
 				}
 
-				require.NotEqual(bm.t, i, nloops-1, "block never managed to sync to node")
+				require.NotEqual(bm.t, i, nloops-1, "block at height %d never managed to sync to node, which is at height %d", epoch, ts.Height())
 				time.Sleep(time.Millisecond * 10)
 			}
 

itests/kit/client.go

@@ -80,7 +80,7 @@ func RunClientTest(t *testing.T, cmds []*lcli.Command, clientNode *TestFullNode)
 	dealStatus := ""
 	for {
 		// client list-deals
-		out = clientCLI.RunCmd("client", "list-deals")
+		out = clientCLI.RunCmd("client", "list-deals", "--show-failed")
 		fmt.Println("list-deals:\n", out)
 
 		lines := strings.Split(out, "\n")

itests/net_test.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"fmt"
 	"testing"
+	"time"
 
 	"github.com/libp2p/go-libp2p/core/network"
 	"github.com/libp2p/go-libp2p/core/peer"
@@ -220,14 +221,17 @@ func TestNetBlockIPAddr(t *testing.T) {
 	firstAddrInfo, _ := firstNode.NetAddrsListen(ctx)
 	secondAddrInfo, _ := secondNode.NetAddrsListen(ctx)
 
-	var secondNodeIPs []string
-
+	secondNodeIPsMap := map[string]struct{}{} // so we can deduplicate
 	for _, addr := range secondAddrInfo.Addrs {
 		ip, err := manet.ToIP(addr)
 		if err != nil {
 			continue
 		}
-		secondNodeIPs = append(secondNodeIPs, ip.String())
+		secondNodeIPsMap[ip.String()] = struct{}{}
+	}
+	var secondNodeIPs []string
+	for s := range secondNodeIPsMap {
+		secondNodeIPs = append(secondNodeIPs, s)
 	}
 
 	// Sanity check that we're not already connected somehow
@@ -243,6 +247,8 @@ func TestNetBlockIPAddr(t *testing.T) {
 	list, err := firstNode.NetBlockList(ctx)
 	require.NoError(t, err)
 
+	fmt.Println(list.IPAddrs)
+	fmt.Println(secondNodeIPs)
 	require.Equal(t, len(list.IPAddrs), len(secondNodeIPs), "expected %d blocked IPs", len(secondNodeIPs))
 	for _, blockedIP := range list.IPAddrs {
 		found := false
@@ -256,10 +262,14 @@ func TestNetBlockIPAddr(t *testing.T) {
 		require.True(t, found, "blocked IP %s is not one of secondNodeIPs", blockedIP)
 	}
 
-	require.Error(t, secondNode.NetConnect(ctx, firstAddrInfo), "shouldn't be able to connect to second node")
-	connectedness, err = secondNode.NetConnectedness(ctx, firstAddrInfo.ID)
-	require.NoError(t, err, "failed to determine connectedness")
-	require.NotEqual(t, connectedness, network.Connected)
+	// a QUIC connection might still succeed when gated, but will be killed right after the handshake
+	_ = secondNode.NetConnect(ctx, firstAddrInfo)
+
+	require.Eventually(t, func() bool {
+		connectedness, err = secondNode.NetConnectedness(ctx, firstAddrInfo.ID)
+		require.NoError(t, err, "failed to determine connectedness")
+		return connectedness != network.Connected
+	}, time.Second*5, time.Millisecond*10)
 
 	// stm: @NETWORK_COMMON_BLOCK_REMOVE_001
 	err = firstNode.NetBlockRemove(ctx, api.NetBlockList{IPAddrs: secondNodeIPs})

itests/paych_api_test.go

@@ -10,7 +10,6 @@ import (
 	cbor "github.com/ipfs/go-ipld-cbor"
 	"github.com/stretchr/testify/require"
 
-	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
 	"github.com/filecoin-project/go-state-types/big"
 	paychtypes "github.com/filecoin-project/go-state-types/builtin/v8/paych"
@@ -19,7 +18,6 @@ import (
 	"github.com/filecoin-project/lotus/blockstore"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/actors/adt"
-	"github.com/filecoin-project/lotus/chain/actors/builtin"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/paych"
 	"github.com/filecoin-project/lotus/chain/actors/policy"
 	"github.com/filecoin-project/lotus/chain/events"
@@ -52,8 +50,7 @@ func TestPaymentChannelsAPI(t *testing.T) {
 		Miner(&miner, &paymentCreator, kit.WithAllSubsystems()).
 		Start().
 		InterconnectAll()
-	bms := ens.BeginMiningMustPost(blockTime)
-	bm := bms[0]
+	ens.BeginMiningMustPost(blockTime)
 
 	waitRecvInSync := func() {
 		// paymentCreator is the block miner, in some cases paymentReceiver may fall behind, so we wait for it to catch up
@@ -197,7 +194,12 @@ func TestPaymentChannelsAPI(t *testing.T) {
 	require.Errorf(t, err, "Expected shortfall error of %d", excessAmt)
 
 	// wait for the settlement period to pass before collecting
-	waitForBlocks(ctx, t, bm, paymentReceiver, receiverAddr, policy.PaychSettleDelay)
+	head, err := paymentReceiver.ChainHead(ctx)
+	require.NoError(t, err)
+
+	settleHeight := head.Height() + policy.PaychSettleDelay + 5
+	paymentReceiver.WaitTillChain(ctx, kit.HeightAtLeast(settleHeight))
+	paymentCreator.WaitTillChain(ctx, kit.HeightAtLeast(settleHeight))
 
 	creatorPreCollectBalance, err := paymentCreator.WalletBalance(ctx, createrAddr)
 	require.NoError(t, err)
@@ -226,31 +228,6 @@ func TestPaymentChannelsAPI(t *testing.T) {
 	require.EqualValues(t, abi.NewTokenAmount(expectedRefund), delta, "did not send correct funds from creator: expected %d, got %d", expectedRefund, delta)
 }
 
-func waitForBlocks(ctx context.Context, t *testing.T, bm *kit.BlockMiner, paymentReceiver kit.TestFullNode, receiverAddr address.Address, count int) {
-	// We need to add null blocks in batches, if we add too many the chain can't sync
-	batchSize := 60
-	for i := 0; i < count; i += batchSize {
-		size := batchSize
-		if i > count {
-			size = count - i
-		}
-
-		// Add a batch of null blocks to advance the chain quicker through finalities.
-		bm.InjectNulls(abi.ChainEpoch(size - 1))
-
-		// Add a real block
-		m, err := paymentReceiver.MpoolPushMessage(ctx, &types.Message{
-			To:    builtin.BurntFundsActorAddr,
-			From:  receiverAddr,
-			Value: types.NewInt(0),
-		}, nil)
-		require.NoError(t, err)
-
-		_, err = paymentReceiver.StateWaitMsg(ctx, m.Cid(), 1, api.LookbackNoLimit, true)
-		require.NoError(t, err)
-	}
-}
-
 func waitForMessage(ctx context.Context, t *testing.T, paymentCreator kit.TestFullNode, msgCid cid.Cid, duration time.Duration, desc string) *api.MsgLookup {
 	ctx, cancel := context.WithTimeout(ctx, duration)
 	defer cancel()

itests/sector_finalize_early_test.go

@@ -31,8 +31,8 @@ func TestDealsWithFinalizeEarly(t *testing.T) {
 	var blockTime = 50 * time.Millisecond
 
 	// We use two miners so that in case the actively tested miner misses PoSt, we still have a blockchain
-	client, miner, _, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC(), kit.MutateSealingConfig(func(sc *config.SealingConfig) { sc.FinalizeEarly = true })) // no mock proofs.
-	ens.InterconnectAll().BeginMining(blockTime)
+	client, miner, poster, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC(), kit.MutateSealingConfig(func(sc *config.SealingConfig) { sc.FinalizeEarly = true })) // no mock proofs.
+	ens.InterconnectAll().BeginMiningMustPost(blockTime, poster)
 	dh := kit.NewDealHarness(t, client, miner, miner)
 
 	ctx := context.Background()

itests/sector_import_simple_test.go

@@ -39,7 +39,9 @@ func TestSectorImportAfterPC2(t *testing.T) {
 	////////
 	// Start a miner node
 
-	client, miner, ens := kit.EnsembleMinimal(t, kit.ThroughRPC())
+	// We use two miners so that in case the actively tested miner misses PoSt, we still have a blockchain
+	client, miner, _, ens := kit.EnsembleOneTwo(t, kit.ThroughRPC())
+
 	ens.InterconnectAll().BeginMining(blockTime)
 
 	ctx := context.Background()

itests/worker_test.go

@@ -1,6 +1,7 @@
 package itests
 
 import (
+	"bytes"
 	"context"
 	"strings"
 	"sync/atomic"
@@ -13,13 +14,17 @@ import (
 
 	"github.com/filecoin-project/go-address"
 	"github.com/filecoin-project/go-state-types/abi"
+	miner11 "github.com/filecoin-project/go-state-types/builtin/v11/miner"
+	"github.com/filecoin-project/go-state-types/network"
 
 	"github.com/filecoin-project/lotus/api"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/types"
 	"github.com/filecoin-project/lotus/itests/kit"
 	"github.com/filecoin-project/lotus/node"
+	"github.com/filecoin-project/lotus/node/config"
 	"github.com/filecoin-project/lotus/node/impl"
+	"github.com/filecoin-project/lotus/node/modules"
 	"github.com/filecoin-project/lotus/node/repo"
 	"github.com/filecoin-project/lotus/storage/paths"
 	sealing "github.com/filecoin-project/lotus/storage/pipeline"
@@ -500,3 +505,228 @@ func TestWorkerName(t *testing.T) {
 
 	require.True(t, found)
 }
+
+// Tests that V1_1 proofs on post workers with faults
+func TestWindowPostV1P1NV20WorkerFault(t *testing.T) {
+	kit.QuietMiningLogs()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	blocktime := 2 * time.Millisecond
+
+	sectors := 2 * 48 * 2
+	var badsector uint64 = 100000
+
+	client, miner, _, ens := kit.EnsembleWorker(t,
+		kit.PresealSectors(sectors), // 2 sectors per partition, 2 partitions in all 48 deadlines
+		kit.GenesisNetworkVersion(network.Version20),
+		kit.ConstructorOpts(
+			node.Override(new(config.ProvingConfig), func() config.ProvingConfig {
+				c := config.DefaultStorageMiner()
+				c.Proving.DisableBuiltinWindowPoSt = true
+				return c.Proving
+			}),
+			node.Override(new(*wdpost.WindowPoStScheduler), modules.WindowPostScheduler(
+				config.DefaultStorageMiner().Fees,
+				config.ProvingConfig{
+					DisableBuiltinWindowPoSt:  true,
+					DisableBuiltinWinningPoSt: false,
+					DisableWDPoStPreChecks:    false,
+				},
+			)),
+			node.Override(new(paths.Store), func(store *paths.Remote) paths.Store {
+				return &badWorkerStorage{
+					Store:       store,
+					badsector:   &badsector,
+					notBadCount: 1,
+				}
+			})),
+		kit.ThroughRPC(),
+		kit.WithTaskTypes([]sealtasks.TaskType{sealtasks.TTGenerateWindowPoSt}),
+		kit.WithWorkerStorage(func(store paths.Store) paths.Store {
+			return &badWorkerStorage{
+				Store:     store,
+				badsector: &badsector,
+			}
+		}))
+
+	bm := ens.InterconnectAll().BeginMining(blocktime)[0]
+
+	maddr, err := miner.ActorAddress(ctx)
+	require.NoError(t, err)
+
+	// wait for sectors to be committed
+	require.Eventually(t, func() bool {
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		parts, err := client.StateMinerPartitions(ctx, maddr, di.Index, types.EmptyTSK)
+		require.NoError(t, err)
+
+		return len(parts) > 1
+	}, 30*time.Second, 100*time.Millisecond)
+
+	// Wait until just before a deadline opens
+	{
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		di = di.NextNotElapsed()
+
+		t.Log("Running one proving period")
+		waitUntil := di.Open + di.WPoStChallengeWindow - di.WPoStChallengeLookback - 1
+		client.WaitTillChain(ctx, kit.HeightAtLeast(waitUntil))
+
+		t.Log("Waiting for post message")
+		bm.Stop()
+	}
+
+	// Remove one sector in the next deadline (so it's skipped)
+	{
+		di, err := client.StateMinerProvingDeadline(ctx, maddr, types.EmptyTSK)
+		require.NoError(t, err)
+
+		parts, err := client.StateMinerPartitions(ctx, maddr, di.Index+1, types.EmptyTSK)
+		require.NoError(t, err)
+		require.Greater(t, len(parts), 0)
+
+		secs := parts[0].AllSectors
+		n, err := secs.Count()
+		require.NoError(t, err)
+		require.Equal(t, uint64(2), n)
+
+		// Drop the sector in first partition
+		sid, err := secs.First()
+		require.NoError(t, err)
+
+		t.Logf("Drop sector %d; dl %d part %d", sid, di.Index, 0)
+
+		atomic.StoreUint64(&badsector, sid)
+		require.NoError(t, err)
+	}
+
+	bm.MineBlocksMustPost(ctx, 2*time.Millisecond)
+
+	mi, err := client.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+	require.NoError(t, err)
+
+	wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+	require.NoError(t, err)
+	en := wact.Nonce
+
+	// wait for a new message to be sent from worker address, it will be a PoSt
+
+waitForProof:
+	for {
+		//stm: @CHAIN_STATE_GET_ACTOR_001
+		wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+		require.NoError(t, err)
+		if wact.Nonce > en {
+			break waitForProof
+		}
+
+		build.Clock.Sleep(blocktime)
+	}
+
+	slm, err := client.StateListMessages(ctx, &api.MessageMatch{To: maddr}, types.EmptyTSK, 0)
+	require.NoError(t, err)
+
+	pmr, err := client.StateSearchMsg(ctx, types.EmptyTSK, slm[0], -1, false)
+	require.NoError(t, err)
+
+	nv, err := client.StateNetworkVersion(ctx, pmr.TipSet)
+	require.NoError(t, err)
+	require.Equal(t, network.Version20, nv)
+
+	require.True(t, pmr.Receipt.ExitCode.IsSuccess())
+
+	slmsg, err := client.ChainGetMessage(ctx, slm[0])
+	require.NoError(t, err)
+
+	var params miner11.SubmitWindowedPoStParams
+	require.NoError(t, params.UnmarshalCBOR(bytes.NewBuffer(slmsg.Params)))
+	require.Equal(t, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1, params.Proofs[0].PoStProof)
+
+	require.Len(t, params.Partitions, 2)
+	sc0, err := params.Partitions[0].Skipped.Count()
+	require.NoError(t, err)
+	require.Equal(t, uint64(1), sc0)
+	sc1, err := params.Partitions[1].Skipped.Count()
+	require.NoError(t, err)
+	require.Equal(t, uint64(0), sc1)
+}
+
+// Tests that V1_1 proofs on post worker
+func TestWindowPostV1P1NV20Worker(t *testing.T) {
+	kit.QuietMiningLogs()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	blocktime := 2 * time.Millisecond
+
+	client, miner, _, ens := kit.EnsembleWorker(t,
+		kit.GenesisNetworkVersion(network.Version20),
+		kit.ConstructorOpts(
+			node.Override(new(config.ProvingConfig), func() config.ProvingConfig {
+				c := config.DefaultStorageMiner()
+				c.Proving.DisableBuiltinWindowPoSt = true
+				return c.Proving
+			}),
+			node.Override(new(*wdpost.WindowPoStScheduler), modules.WindowPostScheduler(
+				config.DefaultStorageMiner().Fees,
+				config.ProvingConfig{
+					DisableBuiltinWindowPoSt:  true,
+					DisableBuiltinWinningPoSt: false,
+					DisableWDPoStPreChecks:    false,
+				},
+			))),
+		kit.ThroughRPC(),
+		kit.WithTaskTypes([]sealtasks.TaskType{sealtasks.TTGenerateWindowPoSt}))
+
+	ens.InterconnectAll().BeginMining(blocktime)
+
+	maddr, err := miner.ActorAddress(ctx)
+	require.NoError(t, err)
+
+	mi, err := client.StateMinerInfo(ctx, maddr, types.EmptyTSK)
+	require.NoError(t, err)
+
+	wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+	require.NoError(t, err)
+	en := wact.Nonce
+
+	// wait for a new message to be sent from worker address, it will be a PoSt
+
+waitForProof:
+	for {
+		//stm: @CHAIN_STATE_GET_ACTOR_001
+		wact, err := client.StateGetActor(ctx, mi.Worker, types.EmptyTSK)
+		require.NoError(t, err)
+		if wact.Nonce > en {
+			break waitForProof
+		}
+
+		build.Clock.Sleep(blocktime)
+	}
+
+	slm, err := client.StateListMessages(ctx, &api.MessageMatch{To: maddr}, types.EmptyTSK, 0)
+	require.NoError(t, err)
+
+	pmr, err := client.StateSearchMsg(ctx, types.EmptyTSK, slm[0], -1, false)
+	require.NoError(t, err)
+
+	nv, err := client.StateNetworkVersion(ctx, pmr.TipSet)
+	require.NoError(t, err)
+	require.Equal(t, network.Version20, nv)
+
+	require.True(t, pmr.Receipt.ExitCode.IsSuccess())
+
+	slmsg, err := client.ChainGetMessage(ctx, slm[0])
+	require.NoError(t, err)
+
+	var params miner11.SubmitWindowedPoStParams
+	require.NoError(t, params.UnmarshalCBOR(bytes.NewBuffer(slmsg.Params)))
+	require.Equal(t, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1, params.Proofs[0].PoStProof)
+}

lib/readerutil/readerutil.go

@@ -0,0 +1,44 @@
+package readerutil
+
+import (
+	"io"
+	"os"
+)
+
+// NewReadSeekerFromReaderAt returns a new io.ReadSeeker from a io.ReaderAt.
+// The returned io.ReadSeeker will read from the io.ReaderAt starting at the
+// given base offset.
+func NewReadSeekerFromReaderAt(readerAt io.ReaderAt, base int64) io.ReadSeeker {
+	return &readSeekerFromReaderAt{
+		readerAt: readerAt,
+		base:     base,
+		pos:      0,
+	}
+}
+
+type readSeekerFromReaderAt struct {
+	readerAt io.ReaderAt
+	base     int64
+	pos      int64
+}
+
+func (rs *readSeekerFromReaderAt) Read(p []byte) (n int, err error) {
+	n, err = rs.readerAt.ReadAt(p, rs.pos+rs.base)
+	rs.pos += int64(n)
+	return n, err
+}
+
+func (rs *readSeekerFromReaderAt) Seek(offset int64, whence int) (int64, error) {
+	switch whence {
+	case io.SeekStart:
+		rs.pos = offset
+	case io.SeekCurrent:
+		rs.pos += offset
+	case io.SeekEnd:
+		return 0, io.ErrUnexpectedEOF
+	default:
+		return 0, os.ErrInvalid
+	}
+
+	return rs.pos, nil
+}

metrics/metrics.go

@@ -61,6 +61,10 @@ var (
 
 	// vm execution
 	ExecutionLane, _ = tag.NewKey("lane")
+
+	// piecereader
+	PRReadType, _ = tag.NewKey("pr_type") // seq / rand
+	PRReadSize, _ = tag.NewKey("pr_size") // small / big
 )
 
 // Measures
@@ -153,8 +157,9 @@ var (
 	SchedCycleOpenWindows                = stats.Int64("sched/assigner_cycle_open_window", "Number of open windows in scheduling cycles", stats.UnitDimensionless)
 	SchedCycleQueueSize                  = stats.Int64("sched/assigner_cycle_task_queue_entry", "Number of task queue entries in scheduling cycles", stats.UnitDimensionless)
 
-	DagStorePRInitCount        = stats.Int64("dagstore/pr_init_count", "PieceReader init count", stats.UnitDimensionless)
-	DagStorePRBytesRequested   = stats.Int64("dagstore/pr_requested_bytes", "PieceReader requested bytes", stats.UnitBytes)
+	DagStorePRInitCount      = stats.Int64("dagstore/pr_init_count", "PieceReader init count", stats.UnitDimensionless)
+	DagStorePRBytesRequested = stats.Int64("dagstore/pr_requested_bytes", "PieceReader requested bytes", stats.UnitBytes)
+
 	DagStorePRBytesDiscarded   = stats.Int64("dagstore/pr_discarded_bytes", "PieceReader discarded bytes", stats.UnitBytes)
 	DagStorePRDiscardCount     = stats.Int64("dagstore/pr_discard_count", "PieceReader discard count", stats.UnitDimensionless)
 	DagStorePRSeekBackCount    = stats.Int64("dagstore/pr_seek_back_count", "PieceReader seek back count", stats.UnitDimensionless)
@@ -162,6 +167,12 @@ var (
 	DagStorePRSeekBackBytes    = stats.Int64("dagstore/pr_seek_back_bytes", "PieceReader seek back bytes", stats.UnitBytes)
 	DagStorePRSeekForwardBytes = stats.Int64("dagstore/pr_seek_forward_bytes", "PieceReader seek forward bytes", stats.UnitBytes)
 
+	DagStorePRAtHitBytes       = stats.Int64("dagstore/pr_at_hit_bytes", "PieceReader ReadAt bytes from cache", stats.UnitBytes)
+	DagStorePRAtHitCount       = stats.Int64("dagstore/pr_at_hit_count", "PieceReader ReadAt from cache hits", stats.UnitDimensionless)
+	DagStorePRAtCacheFillCount = stats.Int64("dagstore/pr_at_cache_fill_count", "PieceReader ReadAt full cache fill count", stats.UnitDimensionless)
+	DagStorePRAtReadBytes      = stats.Int64("dagstore/pr_at_read_bytes", "PieceReader ReadAt bytes read from source", stats.UnitBytes)    // PRReadSize tag
+	DagStorePRAtReadCount      = stats.Int64("dagstore/pr_at_read_count", "PieceReader ReadAt reads from source", stats.UnitDimensionless) // PRReadSize tag
+
 	// splitstore
 	SplitstoreMiss                  = stats.Int64("splitstore/miss", "Number of misses in hotstre access", stats.UnitDimensionless)
 	SplitstoreCompactionTimeSeconds = stats.Float64("splitstore/compaction_time", "Compaction time in seconds", stats.UnitSeconds)
@@ -487,6 +498,7 @@ var (
 	DagStorePRBytesRequestedView = &view.View{
 		Measure:     DagStorePRBytesRequested,
 		Aggregation: view.Sum(),
+		TagKeys:     []tag.Key{PRReadType},
 	}
 	DagStorePRBytesDiscardedView = &view.View{
 		Measure:     DagStorePRBytesDiscarded,
@@ -513,6 +525,29 @@ var (
 		Aggregation: view.Sum(),
 	}
 
+	DagStorePRAtHitBytesView = &view.View{
+		Measure:     DagStorePRAtHitBytes,
+		Aggregation: view.Sum(),
+	}
+	DagStorePRAtHitCountView = &view.View{
+		Measure:     DagStorePRAtHitCount,
+		Aggregation: view.Count(),
+	}
+	DagStorePRAtCacheFillCountView = &view.View{
+		Measure:     DagStorePRAtCacheFillCount,
+		Aggregation: view.Count(),
+	}
+	DagStorePRAtReadBytesView = &view.View{
+		Measure:     DagStorePRAtReadBytes,
+		Aggregation: view.Sum(),
+		TagKeys:     []tag.Key{PRReadSize},
+	}
+	DagStorePRAtReadCountView = &view.View{
+		Measure:     DagStorePRAtReadCount,
+		Aggregation: view.Count(),
+		TagKeys:     []tag.Key{PRReadSize},
+	}
+
 	// splitstore
 	SplitstoreMissView = &view.View{
 		Measure:     SplitstoreMiss,
@@ -779,6 +814,11 @@ var MinerNodeViews = append([]*view.View{
 	DagStorePRSeekForwardCountView,
 	DagStorePRSeekBackBytesView,
 	DagStorePRSeekForwardBytesView,
+	DagStorePRAtHitBytesView,
+	DagStorePRAtHitCountView,
+	DagStorePRAtCacheFillCountView,
+	DagStorePRAtReadBytesView,
+	DagStorePRAtReadCountView,
 }, DefaultViews...)
 
 var GatewayNodeViews = append([]*view.View{

miner/miner.go

@@ -324,7 +324,7 @@ minerLoop:
 					"block-time", btime, "time", build.Clock.Now(), "difference", build.Clock.Since(btime))
 			}
 
-			if err := m.sf.MinedBlock(ctx, b.Header, base.TipSet.Height()+base.NullRounds); err != nil {
+			if _, err = m.sf.MinedBlock(ctx, b.Header, base.TipSet.Height()+base.NullRounds); err != nil {
 				log.Errorf("<!!> SLASH FILTER ERROR: %s", err)
 				if os.Getenv("LOTUS_MINER_NO_SLASHFILTER") != "_yes_i_know_i_can_and_probably_will_lose_all_my_fil_and_power_" {
 					continue

node/builder.go

@@ -89,6 +89,8 @@ const (
 
 	// health checks
 	CheckFDLimit
+	CheckFvmConcurrency
+	LegacyMarketsEOL
 
 	// libp2p
 	PstoreAddSelfKeysKey
@@ -164,6 +166,7 @@ func defaults() []Option {
 		Override(new(dtypes.NodeStartTime), FromVal(dtypes.NodeStartTime(time.Now()))),
 
 		Override(CheckFDLimit, modules.CheckFdLimit(build.DefaultFDLimit)),
+		Override(CheckFvmConcurrency, modules.CheckFvmConcurrency()),
 
 		Override(new(system.MemoryConstraints), modules.MemoryConstraints),
 		Override(InitMemoryWatchdog, modules.MemoryWatchdog),

node/builder_miner.go

@@ -147,6 +147,10 @@ func ConfigStorageMiner(c interface{}) Option {
 		),
 
 		If(cfg.Subsystems.EnableMarkets,
+
+			// Alert that legacy-markets is being deprecated
+			Override(LegacyMarketsEOL, modules.LegacyMarketsEOL),
+
 			// Markets
 			Override(new(dtypes.StagingBlockstore), modules.StagingBlockstore),
 			Override(new(dtypes.StagingGraphsync), modules.StagingGraphsync(cfg.Dealmaking.SimultaneousTransfersForStorage, cfg.Dealmaking.SimultaneousTransfersForStoragePerClient, cfg.Dealmaking.SimultaneousTransfersForRetrieval)),

node/config/def.go

@@ -57,6 +57,10 @@ func defCommon() Common {
 			ListenAddresses: []string{
 				"/ip4/0.0.0.0/tcp/0",
 				"/ip6/::/tcp/0",
+				"/ip4/0.0.0.0/udp/0/quic-v1",
+				"/ip6/::/udp/0/quic-v1",
+				"/ip4/0.0.0.0/udp/0/quic-v1/webtransport",
+				"/ip6/::/udp/0/quic-v1/webtransport",
 			},
 			AnnounceAddresses:   []string{},
 			NoAnnounceAddresses: []string{},

node/config/doc_gen.go

@@ -1107,15 +1107,7 @@ required to have expiration of at least the soonest-ending deal`,
 			Name: "MinTargetUpgradeSectorExpiration",
 			Type: "uint64",
 
-			Comment: `When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-be selected based on lowest initial pledge.
-
-Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-
-Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-initial pledge - upgrade sectors will always be chosen based on longest expiration`,
+			Comment: `DEPRECATED: Target expiration is no longer used`,
 		},
 		{
 			Name: "CommittedCapacitySectorLifetime",

node/config/types.go

@@ -352,15 +352,7 @@ type SealingConfig struct {
 	// required to have expiration of at least the soonest-ending deal
 	MinUpgradeSectorExpiration uint64
 
-	// When set to a non-zero value, minimum number of epochs until sector expiration above which upgrade candidates will
-	// be selected based on lowest initial pledge.
-	//
-	// Target sector expiration is calculated by looking at the input deal queue, sorting it by deal expiration, and
-	// selecting N deals from the queue up to sector size. The target expiration will be Nth deal end epoch, or in case
-	// where there weren't enough deals to fill a sector, DealMaxDuration (540 days = 1555200 epochs)
-	//
-	// Setting this to a high value (for example to maximum deal duration - 1555200) will disable selection based on
-	// initial pledge - upgrade sectors will always be chosen based on longest expiration
+	// DEPRECATED: Target expiration is no longer used
 	MinTargetUpgradeSectorExpiration uint64
 
 	// CommittedCapacitySectorLifetime is the duration a Committed Capacity (CC) sector will