Commit Graph

636 Commits

Author SHA1 Message Date
Mikers
4f0e2f3276 Added CPU profiling option to lotus-shed
This commit includes the following updates:

- Added a new 'pprof' flag to the command-line arguments in main.go. This flag allows the user to specify a file name for writing CPU profile information, enabling performance analysis.

- Implemented functionality to start CPU profiling in the 'Before' function. If a file name is provided via the 'pprof' flag, the function creates the specified file and begins CPU profiling.

- Added an 'After' function to stop CPU profiling once the program execution is finished.

These changes provide a way to analyze the CPU usage of the 'lotus-shed' application, which can be valuable for performance tuning and optimization.
2023-08-02 01:20:06 +00:00
Mikers
1eca520076 Added support for Filecoin network version 21 (v12 state types)
This commit includes the following updates:

- Imported v12 of the filecoin state types in migrations.go. This allows the system to interpret and operate on the latest structure of filecoin state information, ensuring compatibility with the latest version of the Filecoin network.

- Updated the 'getMigrationFuncsForNetwork' function to include a case for network.Version21, enabling the use of the UpgradeActorsV12 and PreUpgradeActorsV12 functions for network migration. These changes ensure the system correctly handles migrations when the network upgrades to version 21.

- Implemented a new 'checkNv21Invariants' function to check the invariants for version 21 of the Filecoin network. This function:

  1. Loads the new state root from the actor store.
  2. Retrieves the actor code IDs for the current state version (v12).
  3. Loads the actor tree from the state root and checks state invariants using the actor code IDs.
  4. Logs any error messages generated during the invariant check.
  5. Finally, prints a message with the time taken to complete the checks.

By checking these invariants, we can ensure the system's state is consistent and valid after migrating to network version 21.
2023-08-01 14:23:55 -10:00
mike seiler
a1bfdfbc04 Refactor state invariant checks for different actor versions
This commit refactors the code in `invariants.go` to support state invariant checks for different actor versions. The code now includes checks for versions 10 and 11 of the actors. The `CheckStateInvariants` function is invoked based on the actor version, passing the appropriate parameters. This change ensures that state invariant checks are performed correctly for different actor versions.

The changes made are as follows:
- Added imports for actor versions 10 and 11.
- Added `case` statements for versions 10 and 11 in the switch statement.
- Invoked `CheckStateInvariants` function with the correct parameters for each version.
- Handled errors returned by the `CheckStateInvariants` function.

These changes enhance the flexibility and compatibility of the code with different actor versions, improving the accuracy of state invariant checks.
2023-06-28 12:50:15 -07:00
Jorropo
6c01310728
chore: migrate to boxo
This migrates everything except the `go-car` librairy: https://github.com/ipfs/boxo/issues/218#issuecomment-1529922103

I didn't migrated everything in the previous release because all the boxo code wasn't compatible with the go-ipld-prime one due to a an in flight (/ aftermath) revert of github.com/ipfs/go-block-format. go-block-format has been unmigrated since slight bellow absolutely everything depends on it that would have required everything to be moved on boxo or everything to optin into using boxo which were all deal breakers for different groups.

This worked fine because lotus's codebase could live hapely on the first multirepo setup however boost is now trying to use boxo's code with lotus's (still on multirepo) setup: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1685022344779649

The alternative would be for boost to write shim types which just forward calls and return with the different interface definitions.

Btw why is that an issue in the first place is because unlike what go's duck typing model suggest interfaces are not transparent https://github.com/golang/go/issues/58112, interfaces are strongly typed but they have implicit narrowing. The issue is if you return an interface from an interface Go does not have a function definition to insert the implicit conversion thus instead the type checker complains you are not returning the right type.

Stubbing types were reverted https://github.com/ipfs/boxo/issues/218#issuecomment-1478650351

Last time I only migrated `go-bitswap` to `boxo/bitswap` because of the security issues and because we never had the interface return an interface problem (we had concrete wrappers where the implicit conversion took place).
2023-06-19 14:45:05 -07:00
Friðrik Ásmundsson
9e4f1a4d23
Add new lotus-shed command for computing eth hash for a given message cid (#10961) 2023-06-06 17:53:04 -05:00
Fridrik Asmundsson
26b4866841 Move back to lotus-shed and add subcategory for indexes 2023-06-03 12:22:25 -05:00
Fridrik Asmundsson
9240776237 fix: improve perf of msgindex backfill 2023-06-01 14:36:05 +00:00
Łukasz Magiera
96fa081ca6
Merge pull request #10913 from filecoin-project/feat/piecereader-perf
feat: piecereader: Allow parallel access
2023-05-30 18:49:41 +02:00
Friðrik Ásmundsson
b269a1ba35
Merge pull request #10877 from filecoin-project/lotus-shed-cancel-context
feat: run lotus-shed commands in context that is cancelled on sigterm
2023-05-27 10:30:52 +00:00
Fridrik Asmundsson
a4c94957e6 Update message when user hits CRTL+C to not imply handling SIGSTOP but SIGTERM/SIGKILL 2023-05-27 10:10:30 +00:00
Łukasz Magiera
4b5a665422 Merge remote-tracking branch 'origin/master' into feat/piecereader-perf 2023-05-26 13:00:42 +02:00
Łukasz Magiera
6c62e6d8e2 fix tests/lint 2023-05-23 12:26:13 +02:00
Jennifer Wang
de0726fcb0 make ledger sign work 2023-05-17 22:01:04 -04:00
Fridrik Asmundsson
0131781808 feat: run lotus-shed commands in context that is cancelled on sigterm 2023-05-16 11:25:57 +00:00
Phi
efdc32f60e Change arg from minerAddress to minerID
Change arg from minerAddress to minerID
2023-05-04 15:21:05 +02:00
Aayush
b4c2c249db chore: deps: update to FVM 3.3.1 2023-05-01 10:37:43 -04:00
Aayush
0d8a3cbaf8 feat: shed tool to report on any consensus mismatches in history 2023-04-28 13:47:24 -04:00
Aayush
33ca0f1f79 shed: migrations: add reminder about continuity testing tool 2023-04-26 09:50:16 -04:00
Aayush
fd7a0e1922 feat: shed: refactor market cron-state command 2023-04-24 11:02:43 -04:00
zenground0
c8e78f7911 Fix lint after merge 2023-04-23 15:43:53 -06:00
Aayush
5a061e4677 Merge branch 'releases' into asr/merge-release-into-master 2023-04-23 14:58:50 -04:00
ZenGround0
327760acff
chore: build: Merge/v22 into 21 for 23 (#10702)
* chore: update ffi to increase execution parallelism

* Don't enforce walking receipt tree during compaction

* fix: build: drop drand incentinet servers

* chore: release lotus v1.20.4

* Apply suggestions from code review

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>

* feat: Introduce nv19 skeleton

Update to go-state-types v0.11.0-alpha-1

Introduce dummy v11 actor bundles

Make new actors adapters

Add upgrade to Upgrade Schedules

make jen

Update to go-state-types v0.11.0-alpha-2

* feat: vm: switch to the new exec trace format (#10372)

This is now "FVM" native. Changes include:

1. Don't treat "trace" messages like off-chain messages. E.g., don't
include CIDs, versions, etc.
2. Include IPLD codecs where applicable.
3. Remove fields that aren't filled by the FVM (timing, some errors,
code locations, etc.).

* feat: implement FIP-0061

* Address review

* Add and test the FIP-0061 migration

* Update actors bundles to fip/20230406

* Update to go-state-types master

* Update to actors v11.0.0-rc1

* - Update go state types
- Keep current expiration defaults on creation, extension some tests
- Update ffi

* ffi experiment

* Integration nv19 migration

- Open splitstore in migration shed tool
- Update state root version

* Post rebase fixup

* Fix

* gen

* nv19 invariant checking

* Try fixig blockstore so bundle is loaded

* Debug

* Fix

* Make butterfly upgrades happen

* Another ffi experiment

* Fix copy paste error

* Actually schedule migration (#10656)

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* Butterfly artifacts

* Set calibration net upgrade height

* Review Response

* Fix state tree version assert

* Quick butterfly upgrade to sanity check (#10660)

* Quick butterfly upgrade to sanity check

* Update butterfly artifacts

* Revert fake fix

* Give butterfly net correct genesis

* Butterfly artifacts

* Give time before upgrade

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* chore:releasepolish v1.22 release (#10666)

* Update butterfly artifacts

* register actors v11

* Update calibration upgrade time

* State inspection shed cmds

* Fix

* make gen

* Fix swallowed errors

* Lint fixup

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* v1.22.0-rc3

* bundle fix

* Feat/expedite nv19 (#10681)

* Update go-state-types

* Modify upgrade schedule and params

* Revert fip 0052

* Update gst

* docsgen

* fast butterfly migration to validate migration

* Correct epoch to match specified date

* Update actors v11

* Update changelog build version

* Update butterfly artifacts

* Fix lotus-miner init to work after upgrade

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* fix:deps:stable ffi for stable release  (#10698)

* Point to stable ffi for stable lotus release

* go mod tidy

---------

Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>

* Update CHANGELOG.md

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>

---------

Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com>
Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: jennijuju <jiayingw703@gmail.com>
2023-04-19 18:40:18 -04:00
Jorropo
fe42d974a2 chore: all: migrate from go-libipfs to boxo
github.com/ipfs/libipfs/blocks was unmigrated to github.com/ipfs/go-block-format due to compatibility issues with the rest of the IPLD stack.
2023-04-18 17:22:18 +02:00
Jennifer Wang
b82726f57f make lint happy 2023-03-31 15:16:24 -04:00
Jennifer Wang
28fdc3abb8 docsgen 2023-03-31 15:13:13 -04:00
Jennifer Wang
4a1eccd447 add unique 2023-03-31 15:05:03 -04:00
Jennifer Wang
77883aa72b add accounts 2023-03-31 15:05:00 -04:00
Jennifer Wang
07c9367467 wording 2023-03-31 15:04:27 -04:00
zenground0
1a5e8603c7 Fix dumb mistake 2023-03-31 15:04:27 -04:00
zenground0
3b0d8fdd29 P0 fix 2023-03-31 15:04:27 -04:00
Jennifer Wang
ec90ccf72e point to the hot store 2023-03-31 15:04:27 -04:00
Jennifer Wang
661d8608bc evm account balances
Please enter the commit message for your changes. Lines starting
2023-03-31 15:04:24 -04:00
Jiaying Wang
caa6dd9c41
Merge pull request #10594 from filecoin-project/phi/v1210-rc2
build: release: v1.21.0-rc2
2023-03-29 15:59:56 -04:00
Aayush
66fc6dc3e5 refactor: stop using deprecated io/ioutil 2023-03-29 15:40:49 -04:00
Aayush
a724a713da refactor: stop using deprecated io/ioutil 2023-03-29 15:38:46 -04:00
Mikers
06294351fa perf: eth: gas estimate set applyTsMessages false (#10546)
* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal>
2023-03-29 14:25:40 -04:00
Mikers
41fce94db4
perf: eth: gas estimate set applyTsMessages false (#10546)
* have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization

* set applyTsMessages = true in CallWithGas call in shed

* update test with new callwithgas api optimization for eth call

* Update chain/stmgr/call.go

Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change

* make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation

---------

Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com>
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal>
2023-03-23 18:27:01 -04:00
Aayush Rajasekaran
4efc29aa64
Merge pull request #10513 from filecoin-project/feat/shed-chainwatch
feat: shed: incoming block-sub chainwatch tool
2023-03-23 17:50:41 -04:00
Łukasz Magiera
e2ff9027f9 shed chainwatch: Appease the linter 2023-03-23 12:33:00 +01:00
vyzo
8abe0ea608
Merge pull request #10452 from filecoin-project/vyzo/feat/chain/msgindex
feat:chain: Message Index
2023-03-22 17:02:34 +02:00
Łukasz Magiera
02db37bf37 feat: shed: incoming block-sub chainwatch tool 2023-03-20 10:37:48 +01:00
vyzo
1676d51484 shed: expand homedir in repo path for msgindex tools 2023-03-16 17:17:26 +02:00
Steven Allen
dcb49dc8ee
refactor: update cache to the new generic version (#10463)
- Adds type safety.
- Reduces allocations.
- Fixes the drand cache (was storing by value, but retrieving by pointer)
2023-03-13 15:29:09 -07:00
vyzo
6deec4c5f4 lint 2023-03-13 17:26:20 +02:00
vyzo
06f93861bc make gen 2023-03-13 17:00:16 +02:00
vyzo
27dc4951eb lotus-shed tools for msgindex 2023-03-13 16:51:21 +02:00
vyzo
1415391988 lint
lint

lint
2023-03-12 15:50:10 +02:00
Travis Person
f641139bf2
Merge pull request #10282 from filecoin-project/feat/cache-migrated-stateroot
feat: stmgr: cache migrated stateroots
2023-03-09 11:47:43 -08:00
Aayush
19f1ad2762 Merge branch 'releases' into asr/merge-release-into-master 2023-03-01 14:18:35 -05:00
Travis Person
814c146626 feat: stmgr: cache migrated stateroots 2023-03-01 16:17:34 +00:00