Commit Graph

157 Commits

Author SHA1 Message Date
Łukasz Magiera
6a0f16b084 feat: sealing: Support nv22 DDO features in the sealing pipeline (#11226)
* Initial work supporting DDO pieces in lotus-miner

* sealing: Update pipeline input to operate on UniversalPiece

* sealing: Update pipeline checks/sealing states to operate on UniversalPiece

* sealing: Make pipeline build with UniversalPiece

* move PieceDealInfo out of api

* make gen

* make sealing pipeline unit tests pass

* fix itest ensemble build

* don't panic in SectorsStatus with deals

* stop linter from complaining about checkPieces

* fix sector import tests

* mod tidy

* sealing: Add logic for (pre)committing DDO sectors

* sealing: state-types with method defs

* DDO non-snap pipeline works(?), DDO Itests

* DDO support in snapdeals pipeline

* make gen

* update actor bundles

* update the gst market fix

* fix: chain: use PreCommitSectorsBatch2 when setting up genesis

* some bug fixes

* integration working changes

* update actor bundles

* Make TestOnboardRawPieceSnap pass

* Appease the linter

* Make deadlines test pass with v12 actors

* Update go-state-types, abstract market DealState

* make gen

* mod tidy, lint fixes

* Fix some more tests

* Bump version in master

Bump version in master

* Make gen

Make gen

* fix sender

* fix: lotus-provider: Fix winning PoSt

* fix: sql Scan cannot write to an object

* Actually show miner-addrs in info-log

Actually show miner-addrs in lotus-provider info-log

* [WIP] feat: Add nv22 skeleton

Addition of Network Version 22 skeleton

* update FFI

* ddo is now nv22

* make gen

* temp actor bundle with ddo

* use working go-state-types

* gst with v13 market migration

* update bundle, builtin.MethodsMiner.ProveCommitSectors2 -> 3

* actually working v13 migration, v13 migration itest

* Address review

* sealing: Correct DDO snap pledge math

* itests: Mixed ddo itest

* pipeline: Fix sectorWeight

* sealing: convert market deals into PAMs in mixed sectors

* sealing: make market to ddo conversion work

* fix lint

* update gst

* Update actors and GST to lastest integ branch

* commit batcher: Update ProveCommitSectors3Params builder logic

* make gen

* use builtin-actors master

* ddo: address review

* itests: Add commd assertions to ddo tests

* make gen

* gst with fixed types

* config knobs for RequireActivationSuccess

* storage: Drop obsolete flaky tasts

---------

Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Aayush <arajasek94@gmail.com>
Co-authored-by: Shrenuj Bansal <shrenuj.bansal@protocol.ai>
Co-authored-by: Phi <orjan.roren@gmail.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com>
Co-authored-by: TippyFlits <james.bluett@protocol.ai>
2024-03-22 07:00:28 +01:00
Andrew Jackson (Ajax)
848c20d15f Merge branch 'master' into feat/sturdypost 2023-11-06 16:10:57 -06:00
Aayush
5e76b05b17 Merge branch 'feat/nv21' into asr/merge-feat-nv21 2023-10-16 11:13:33 -04:00
Andrew Jackson (Ajax)
ea035f4a7f lint, db timout, cfg rm 2023-09-29 11:56:10 -05:00
Andrew Jackson (Ajax)
a520ee85d6 master-to-sturdy 2023-09-21 10:37:02 -05:00
Fridrik Asmundsson
861731ee7c Add support to easily benchmark binary for understanding their latencies/histogram 2023-09-14 14:17:42 +00:00
zenground0
4d8f8d2600 Review Response 2023-07-20 16:59:11 -06:00
Łukasz Magiera
7adf8237f6 add synthPoRep support to bench simple 2023-07-18 10:27:40 -07:00
zenground0
38b21a713a Lint 2023-07-13 08:41:34 -06:00
Andrew Jackson (Ajax)
b43665a3c1 synthPoRep in PreferredSeal..() 2023-07-13 15:47:58 +02:00
zenground0
6eadc0ee57 gen 2023-07-13 07:37:51 -06:00
zenground0
eaf0200fe1 amt bench in good state 2023-07-12 23:58:02 -06:00
Fridrik Asmundsson
4d90d0475f Upgrade urfave dependency which now supports DisableSliceFlagSeparator flag 2023-06-07 15:47:58 -05:00
Fridrik Asmundsson
e1b69f84d6 Cleanup after removing urface upgrade 2023-05-27 10:55:10 +00:00
Fridrik Asmundsson
78d7ccd391 Add new RPC stress testing tool (lotus-bench rpc) with rich reporting
This benchmark is designed to stress test the rpc methods of a lotus node so that we can simulate real world usage and measure the performance of rpc methods on the node.

This benchmark has the following features:
* Can query each method both sequentially and concurrently
* Supports rate limiting
* Can query multiple different endpoints at once (supporting different concurrency level and rate limiting for each method)
* Gives a nice reporting summary of the stress testing of each method (including latency distribution, histogram and more)
* Easy to use

To use this benchmark you must specify the rpc methods you want to test using the --method options, the format of it is:

  --method=NAME[:CONCURRENCY][:QPS][:PARAMS] where only METHOD is required.

Here are some real examples:
  lotus-bench rpc --method='eth_chainId' // run eth_chainId with default concurrency and qps
  lotus-bench rpc --method='eth_chainId:3'  // override concurrency to 3
  lotus-bench rpc --method='eth_chainId::100' // override to 100 qps while using default concurrency
  lotus-bench rpc --method='eth_chainId:3:100' // run using 3 workers but limit to 100 qps
  lotus-bench rpc --method='eth_getTransactionCount:::["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run using optional params while using default concurrency and qps
  lotus-bench rpc --method='eth_chainId' --method='eth_getTransactionCount:10:0:["0xd4c70007F3F502f212c7e6794b94C06F36173B36", "latest"]' // run multiple methods at once`,

Fixes: https://github.com/filecoin-project/lotus/issues/10752
2023-05-27 10:55:10 +00:00
ZenGround0
327760acff
chore: build: Merge/v22 into 21 for 23 (#10702)
* chore: update ffi to increase execution parallelism

* Don't enforce walking receipt tree during compaction

* fix: build: drop drand incentinet servers

* chore: release lotus v1.20.4

* Apply suggestions from code review

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>

* feat: Introduce nv19 skeleton

Update to go-state-types v0.11.0-alpha-1

Introduce dummy v11 actor bundles

Make new actors adapters

Add upgrade to Upgrade Schedules

make jen

Update to go-state-types v0.11.0-alpha-2

* feat: vm: switch to the new exec trace format (#10372)

This is now "FVM" native. Changes include:

1. Don't treat "trace" messages like off-chain messages. E.g., don't
include CIDs, versions, etc.
2. Include IPLD codecs where applicable.
3. Remove fields that aren't filled by the FVM (timing, some errors,
code locations, etc.).

* feat: implement FIP-0061

* Address review

* Add and test the FIP-0061 migration

* Update actors bundles to fip/20230406

* Update to go-state-types master

* Update to actors v11.0.0-rc1

* - Update go state types
- Keep current expiration defaults on creation, extension some tests
- Update ffi

* ffi experiment

* Integration nv19 migration

- Open splitstore in migration shed tool
- Update state root version

* Post rebase fixup

* Fix

* gen

* nv19 invariant checking

* Try fixig blockstore so bundle is loaded

* Debug

* Fix

* Make butterfly upgrades happen

* Another ffi experiment

* Fix copy paste error

* Actually schedule migration (#10656)

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* Butterfly artifacts

* Set calibration net upgrade height

* Review Response

* Fix state tree version assert

* Quick butterfly upgrade to sanity check (#10660)

* Quick butterfly upgrade to sanity check

* Update butterfly artifacts

* Revert fake fix

* Give butterfly net correct genesis

* Butterfly artifacts

* Give time before upgrade

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* chore:releasepolish v1.22 release (#10666)

* Update butterfly artifacts

* register actors v11

* Update calibration upgrade time

* State inspection shed cmds

* Fix

* make gen

* Fix swallowed errors

* Lint fixup

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* v1.22.0-rc3

* bundle fix

* Feat/expedite nv19 (#10681)

* Update go-state-types

* Modify upgrade schedule and params

* Revert fip 0052

* Update gst

* docsgen

* fast butterfly migration to validate migration

* Correct epoch to match specified date

* Update actors v11

* Update changelog build version

* Update butterfly artifacts

* Fix lotus-miner init to work after upgrade

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* fix:deps:stable ffi for stable release  (#10698)

* Point to stable ffi for stable lotus release

* go mod tidy

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* Update CHANGELOG.md

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>

---------

Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: jennijuju <jiayingw703@gmail.com>
2023-04-19 18:40:18 -04:00
Aayush
a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
Phi
c6f69946be fix: bench: remove unused env
fix: bench: remove unused environment variable.
2023-01-02 09:07:45 +01:00
Aayush
32670e810c chore: refactor: rename NewestNetworkVersion 2022-09-21 15:48:51 -04:00
Łukasz Magiera
58416d0881 storage: Don't depend on specs-storage, move to storiface 2022-06-17 13:36:33 +02:00
Łukasz Magiera
05cdeb80c3 chore: remove redundant import prefixes 2022-06-15 12:06:22 +02:00
Łukasz Magiera
a9600b8a6f storage: Move extern/sector-storage to storage/sealer 2022-06-14 20:03:38 +02:00
Łukasz Magiera
e65fae28de chore: fix imports 2022-06-14 17:00:51 +02:00
Aayush
b28c11a57d Merge branch 'feat/nv16' 2022-06-03 14:01:49 -04:00
Łukasz Magiera
1960a7726b bench: Simple distributed post, print perf 2022-05-30 15:44:05 +02:00
Aayush
8cca9b1970 Use new go-state-types accessors 2022-05-17 15:21:27 -04:00
eben.xie
e2cbad6ff4 [Describe]: when excute cmd "lotus-bench sealing" without "benchmark-existing-sectorbuilder", panic will occur
[BugFix]:

[FeatureAdd]:

[CodeReview]:

[ModifyDesc]:

[Author]:

[BugID]:
2022-02-23 10:49:53 +08:00
Aayush Rajasekaran
d645c5fbab Remove unnecessary params from VerifyWinningPost 2022-01-11 12:06:39 -05:00
zenground0
33f2d24f54 Snap Deals Integration
- FSM handles the actual cc upgrade process including error states
- PoSting (winning and window) works over upgraded and upgrading sectors
- Integration test and changes to itest framework to reduce flakes
- Update CLI to handle new upgrade
- Update dependencies
2022-01-10 15:39:38 +05:30
Łukasz Magiera
506f39b294 WIP: Integrate FIP0013 2021-05-11 22:10:29 -04:00
Łukasz Magiera
b1dfd41c19
Merge pull request #4950 from zgfzgf/feat-bench-policy
delete bench policy
2021-01-30 12:34:40 +01:00
Frank
fdc46ad5df update json format 2021-01-06 12:20:33 +08:00
Frank
f24d401b61 add bench env print 2020-12-18 17:57:32 +08:00
zgfzgf
e197038b9f delete policy useless code 2020-11-21 10:14:13 +08:00
zgfzgf
549afbaa5f add policy 8m and 512m 2020-11-21 10:04:07 +08:00
zgfzgf
d033042d6b update code 2020-11-20 23:20:26 +08:00
zgfzgf
7d4bc90d42 solve merge conflicts 2020-11-20 22:34:15 +08:00
Łukasz Magiera
915c5e0fdd Get most tests to pass 2020-11-16 19:03:30 +01:00
zgfzgf
3e7efad0fd modify lint error 2020-11-03 21:28:01 +08:00
zgfzgf
3c2cd506ec modify merge lotus-bench 2020-11-03 19:56:04 +08:00
Łukasz Magiera
660236b224 Merge remote-tracking branch 'origin/master' into feat/async-restartable-workers 2020-10-23 23:25:35 +02:00
Steven Allen
4e730b5ec8 port to v2 imports 2020-10-21 12:16:23 -07:00
zgfzgf
bc1f18b73a lotus bench optimize 2020-10-11 18:55:12 +08:00
zgfzgf
c3a405eac7 add lotus bench print 2020-10-11 18:26:27 +08:00
zgfzgf
e4209d86f6 lotus bench add ticket-preimage 2020-10-11 17:20:55 +08:00
zgfzgf
fcb9dea501 add sum sealing time 2020-10-11 16:31:39 +08:00
zgfzgf
c810b157cf lotus bench delete ix 2020-10-11 11:39:27 +08:00
zgfzgf
c458f6e826 lotus bench replace abi.SectorID with sid 2020-10-11 11:25:19 +08:00
zgfzgf
c99e223cb1 lotus bench change and add print 2020-10-11 10:48:09 +08:00
zgfzgf
5c27d6aeee lotus-bench UnsealPiece SectorID.Number error 2020-10-11 10:12:06 +08:00