From 441f5995d30d32cb34ef98f49aad143fcb6e5133 Mon Sep 17 00:00:00 2001 From: "Andrew Jackson (Ajax)" Date: Sat, 4 May 2024 03:35:30 -0500 Subject: [PATCH] fix: curio: node UI & darwin gpu count (#11950) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: https://github.com/filecoin-project/go-state-types/pull/260 Ref: https://github.com/whyrusleeping/cbor-gen/pull/96 Ref: https://github.com/filecoin-project/go-amt-ipld/pull/90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan Co-authored-by: Rod Vagg Co-authored-by: Steven Allen * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * https://github.com/filecoin-project/lotus/pull/11723#discussion_r1534728607 * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from https://github.com/filecoin-project/lotus/pull/11723 Closes: https://github.com/filecoin-project/lotus/issues/11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug Co-authored-by: parthshah1 Co-authored-by: Piotr Galar Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan Co-authored-by: Rod Vagg Co-authored-by: Steven Allen Co-authored-by: Masih H. Derkani Co-authored-by: Lee Co-authored-by: Andrew Jackson (Ajax) Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera Co-authored-by: Łukasz Magiera Co-authored-by: Aarsh Shah * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug Co-authored-by: Łukasz Magiera Co-authored-by: Łukasz Magiera Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: parthshah1 Co-authored-by: Piotr Galar Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan Co-authored-by: Rod Vagg Co-authored-by: Steven Allen Co-authored-by: Masih H. Derkani Co-authored-by: Lee Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Aarsh Shah --- curiosrc/web/api/config/config.go | 23 +++++++++++++++ curiosrc/web/hapi/simpleinfo.go | 17 +++++++---- curiosrc/web/hapi/web/cluster_machines.gohtml | 3 ++ curiosrc/web/hapi/web/node_info.gohtml | 8 +++--- curiosrc/web/hapi/web/root.gohtml | 5 ++-- curiosrc/web/static/config/index.html | 28 +++++++++++++++++++ curiosrc/web/static/index.html | 17 ++++++----- lib/harmony/resources/getGPU_darwin.go | 2 +- storage/paths/local.go | 6 ++-- 9 files changed, 86 insertions(+), 23 deletions(-) diff --git a/curiosrc/web/api/config/config.go b/curiosrc/web/api/config/config.go index 6f9598f7f..414ee7e6f 100644 --- a/curiosrc/web/api/config/config.go +++ b/curiosrc/web/api/config/config.go @@ -11,12 +11,15 @@ import ( "github.com/BurntSushi/toml" "github.com/gorilla/mux" "github.com/invopop/jsonschema" + logging "github.com/ipfs/go-log/v2" "github.com/filecoin-project/lotus/cmd/curio/deps" "github.com/filecoin-project/lotus/curiosrc/web/api/apihelper" "github.com/filecoin-project/lotus/node/config" ) +var log = logging.Logger("curio/web/config") + type cfg struct { *deps.Deps } @@ -30,9 +33,29 @@ func Routes(r *mux.Router, deps *deps.Deps) { // At edit.html: r.Methods("GET").Path("/schema").HandlerFunc(getSch) r.Methods("GET").Path("/layers/{layer}").HandlerFunc(c.getLayer) + r.Methods("POST").Path("/addlayer").HandlerFunc(c.addLayer) r.Methods("POST").Path("/layers/{layer}").HandlerFunc(c.setLayer) r.Methods("GET").Path("/default").HandlerFunc(c.def) } + +func (c *cfg) addLayer(w http.ResponseWriter, r *http.Request) { + var layer struct { + Name string + } + apihelper.OrHTTPFail(w, json.NewDecoder(r.Body).Decode(&layer)) + ct, err := c.DB.Exec(context.Background(), `INSERT INTO harmony_config (title, config) VALUES ($1, $2)`, layer.Name, "") + apihelper.OrHTTPFail(w, err) + if ct != 1 { + w.WriteHeader(http.StatusConflict) + _, err = w.Write([]byte("Layer already exists")) + if err != nil { + log.Errorf("Failed to write response: %s", err) + } + return + } + w.WriteHeader(200) +} + func getSch(w http.ResponseWriter, r *http.Request) { ref := jsonschema.Reflector{ Mapper: func(i reflect.Type) *jsonschema.Schema { diff --git a/curiosrc/web/hapi/simpleinfo.go b/curiosrc/web/hapi/simpleinfo.go index 11877db15..0e9847c4b 100644 --- a/curiosrc/web/hapi/simpleinfo.go +++ b/curiosrc/web/hapi/simpleinfo.go @@ -13,6 +13,7 @@ import ( "text/template" "time" + "github.com/dustin/go-humanize" "github.com/gorilla/mux" "github.com/samber/lo" "golang.org/x/xerrors" @@ -517,7 +518,10 @@ type machineSummary struct { ID int64 SinceContact string - RecentTasks []*machineRecentTask + RecentTasks []*machineRecentTask + Cpu int + RamHumanized string + Gpu int } type taskSummary struct { @@ -580,7 +584,7 @@ func (a *app) clusterMachineSummary(ctx context.Context) ([]machineSummary, erro } // Then machine summary - rows, err := a.db.Query(ctx, "SELECT id, host_and_port, last_contact FROM harmony_machines order by host_and_port asc") + rows, err := a.db.Query(ctx, "SELECT id, host_and_port, CURRENT_TIMESTAMP - last_contact AS last_contact, cpu, ram, gpu FROM harmony_machines order by host_and_port asc") if err != nil { return nil, err // Handle error } @@ -589,13 +593,14 @@ func (a *app) clusterMachineSummary(ctx context.Context) ([]machineSummary, erro var summaries []machineSummary for rows.Next() { var m machineSummary - var lastContact time.Time + var lastContact time.Duration + var ram int64 - if err := rows.Scan(&m.ID, &m.Address, &lastContact); err != nil { + if err := rows.Scan(&m.ID, &m.Address, &lastContact, &m.Cpu, &ram, &m.Gpu); err != nil { return nil, err // Handle error } - - m.SinceContact = time.Since(lastContact).Round(time.Second).String() + m.SinceContact = lastContact.Round(time.Second).String() + m.RamHumanized = humanize.Bytes(uint64(ram)) // Add recent tasks if ts, ok := taskSummaries[m.Address]; ok { diff --git a/curiosrc/web/hapi/web/cluster_machines.gohtml b/curiosrc/web/hapi/web/cluster_machines.gohtml index 5fb18b52c..6c04b2871 100644 --- a/curiosrc/web/hapi/web/cluster_machines.gohtml +++ b/curiosrc/web/hapi/web/cluster_machines.gohtml @@ -3,6 +3,9 @@ {{.Address}} {{.ID}} + {{.Cpu}} + {{.RamHumanized}} + {{.Gpu}} {{.SinceContact}} {{range .RecentTasks}} {{.TaskName}}:{{.Success}}{{if ne 0 .Fail}}({{.Fail}}){{end}} diff --git a/curiosrc/web/hapi/web/node_info.gohtml b/curiosrc/web/hapi/web/node_info.gohtml index 16f60a475..98e56afd8 100644 --- a/curiosrc/web/hapi/web/node_info.gohtml +++ b/curiosrc/web/hapi/web/node_info.gohtml @@ -1,6 +1,6 @@ {{define "node_info"}}

Info

- +
@@ -22,7 +22,7 @@
Host ID

Storage

- +
@@ -56,7 +56,7 @@

Tasks

Running

-
ID Type
+
@@ -73,7 +73,7 @@ {{end}}
ID Task

Recently Finished

- +
diff --git a/curiosrc/web/hapi/web/root.gohtml b/curiosrc/web/hapi/web/root.gohtml index 114db6462..8fe3ba71a 100644 --- a/curiosrc/web/hapi/web/root.gohtml +++ b/curiosrc/web/hapi/web/root.gohtml @@ -1,10 +1,11 @@ {{define "root"}} +{{.PageTitle}} - - + + diff --git a/curiosrc/web/static/config/index.html b/curiosrc/web/static/config/index.html index 769929e53..28a11f6d9 100644 --- a/curiosrc/web/static/config/index.html +++ b/curiosrc/web/static/config/index.html @@ -22,6 +22,7 @@ this.loadData(); this.message = this.readCookie('message'); this.eraseCookie('message'); + this.addName = ''; } static get styles() { return [css` @@ -46,6 +47,13 @@ `)}
ID Task
+ + +
+ + To delete a layer, use ysqlsh to issue the following command:
+ DELETE FROM curio.harmony_config WHERE title = 'my_layer_name'; + `; } loadData() { @@ -77,6 +85,26 @@ eraseCookie(name) { document.cookie = name + '=; expires=Thu, 01 Jan 1970 00:00:00 UTC; path=/;'; } + + updateName(e) { + this.addName = e.target.value; + } + addLayer() { + // get a name + var v = this.addName; + if (v === '') { + alert('Error: Layer name cannot be empty'); + return; + } + + axios.post('/api/config/addlayer', { name: v }) + .then(response => { + window.location.href = '/config/edit.html?layer=' + v; + }) + .catch(error => { + alert('Error adding layer:', error); + }); + } }); diff --git a/curiosrc/web/static/index.html b/curiosrc/web/static/index.html index 0d734d6b7..57c6d4856 100644 --- a/curiosrc/web/static/index.html +++ b/curiosrc/web/static/index.html @@ -60,7 +60,7 @@
-
+

Chain Connectivity

@@ -71,7 +71,7 @@
-
+

Cluster Machines

@@ -79,6 +79,9 @@ + + + @@ -93,7 +96,7 @@
-
+

PoRep Pipeline

Host IDCPUsRAMGPUs Last Contact Tasks (24h)
@@ -120,7 +123,7 @@
-
+

Actor Summary

@@ -133,7 +136,7 @@ - + @@ -147,7 +150,7 @@
-
+

Recently Finished Tasks

Balance Available WorkerWinsWins
@@ -174,7 +177,7 @@
-
+

Cluster Tasks

diff --git a/lib/harmony/resources/getGPU_darwin.go b/lib/harmony/resources/getGPU_darwin.go index a9c0a33cd..b977ec638 100644 --- a/lib/harmony/resources/getGPU_darwin.go +++ b/lib/harmony/resources/getGPU_darwin.go @@ -4,5 +4,5 @@ package resources func getGPUDevices() float64 { - return 10000.0 // unserious value intended for non-production use. + return 1.0 // likely-true value intended for non-production use. } diff --git a/storage/paths/local.go b/storage/paths/local.go index 9d662e88a..07223ad53 100644 --- a/storage/paths/local.go +++ b/storage/paths/local.go @@ -462,9 +462,9 @@ func (st *Local) reportStorage(ctx context.Context) { } } -func (st *Local) Reserve(ctx context.Context, sid storiface.SectorRef, ft storiface.SectorFileType, storageIDs storiface.SectorPaths, overheadTab map[storiface.SectorFileType]int, minFreePercentage float64) (userRelease func(), err error) { - var ssize abi.SectorSize - ssize, err = sid.ProofType.SectorSize() +func (st *Local) Reserve(ctx context.Context, sid storiface.SectorRef, ft storiface.SectorFileType, + storageIDs storiface.SectorPaths, overheadTab map[storiface.SectorFileType]int, minFreePercentage float64) (func(), error) { + ssize, err := sid.ProofType.SectorSize() if err != nil { return nil, err }