Design scheme for fetching yarn dependencies from arbitrary registries #64

New Issue

dboreham · 2022-12-13T13:45:54Z

dboreham commented

2022-12-13 13:45:54 +00:00

Currently yarn stores the host name used to fetch dependencies in yarn.lock. Since yarn.lock is in the git repo, this effectively prevents decentralized builds (see: https://github.com/yarnpkg/yarn/issues/2566 and https://github.com/yarnpkg/rfcs/pull/64). We need to devise some method for overriding the host name in yarn.lock so our users can build packages using their npm registry of choice.

Currently yarn stores the host name used to fetch dependencies in `yarn.lock`. Since `yarn.lock` is in the git repo, this effectively prevents decentralized builds (see: https://github.com/yarnpkg/yarn/issues/2566 and https://github.com/yarnpkg/rfcs/pull/64). We need to devise some method for overriding the host name in `yarn.lock` so our users can build packages using their npm registry of choice.

dboreham commented

2022-12-13 14:41:10 +00:00

Trying this: https://github.com/kiegroup/lock-treatment-tool

dboreham commented

2022-12-13 15:15:45 +00:00

Some experimentation:

Redhat's lock-treatment-tool doesn't do what we need because its behavior can't be made scope-specific. Perhaps we can upgrade it to support scoping.

Running yarn remove @cerc-io/package then yarn add @cerc-io/package@version doesn't work because it updates the versions of transitive dependencies of "package".

Some experimentation: Redhat's lock-treatment-tool doesn't do what we need because its behavior can't be made scope-specific. Perhaps we can upgrade it to support scoping. Running `yarn remove @cerc-io/package` then `yarn add @cerc-io/package@version` doesn't work because it updates the versions of transitive dependencies of "package".

dboreham commented

2022-12-13 17:25:59 +00:00

Trying this: https://stackoverflow.com/questions/49501749/is-it-possible-to-ignore-the-dependency-hash-validation-of-just-one-module-or-r

That didn't work either.

Trying this: https://stackoverflow.com/questions/49501749/is-it-possible-to-ignore-the-dependency-hash-validation-of-just-one-module-or-r That didn't work either.

dboreham commented

2022-12-13 17:54:37 +00:00

It appears that we need to edit yarn.lock to replace whatever repository URL and checksum is there with the correct info for our local registry. Luckily yarn info can show us that info:

  dist: {
    integrity: 'sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==',
    shasum: 'd5b7b7e96a4416117731dafc325c2b5c61fb5dc2',
    tarball: 'http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz'
  }

which we can use to make an edit like this:

diff --git a/yarn.lock b/yarn.lock
index 5066ad9..3b41ba5 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -4,8 +4,8 @@

 "@cerc-io/laconic-sdk@0.1.4":
   version "0.1.4"
-  resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9"
-  integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ==
+  resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2"
+  integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==
   dependencies:
     "@cosmjs/amino" "^0.28.1"

Some testing suggests this works. Next step is to create some tooling to automate the process.

It appears that we need to edit `yarn.lock` to replace whatever repository URL and checksum is there with the correct info for our local registry. Luckily `yarn info` can show us that info: ``` dist: { integrity: 'sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==', shasum: 'd5b7b7e96a4416117731dafc325c2b5c61fb5dc2', tarball: 'http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz' } ``` which we can use to make an edit like this: ``` diff --git a/yarn.lock b/yarn.lock index 5066ad9..3b41ba5 100644 --- a/yarn.lock +++ b/yarn.lock @@ -4,8 +4,8 @@ "@cerc-io/laconic-sdk@0.1.4": version "0.1.4" - resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9" - integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ== + resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2" + integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw== dependencies: "@cosmjs/amino" "^0.28.1" ``` Some testing suggests this works. Next step is to create some tooling to automate the process.

dboreham commented

2022-12-14 14:23:50 +00:00

Success. This works:

yarn_info_output=$(yarn info --json @cerc-io/laconic-sdk@0.1.4 2>/dev/null)
package_tarball=$(echo $yarn_info_output | jq -r .data.dist.tarball)
package_integrity=$(echo $yarn_info_output | jq -r .data.dist.integrity)
package_shasum=$(echo $yarn_info_output | jq -r .data.dist.shasum)
package_resolved=${package_tarball}#${package_shasum}
escaped_package_resolved=$(printf '%s\n' "$package_resolved" | sed -e 's/[\/&]/\\&/g')
if [ -n "$CERC_SCRIPT_VERBOSE" ]; then
    echo "Tarball: ${package_tarball}"
    echo "Integrity: ${package_integrity}"
    echo "Shasum: ${package_shasum}"
    echo "Resolved: ${package_resolved}"
fi
sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^\([[:space:]]\{1,\}resolved \).*$/\1'\"${escaped_package_resolved}\"'/' yarn.lock
sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^\([[:space:]]\{1,\}integrity \).*$/\1'${package_integrity}'/' yarn.lock

diff --git a/yarn.lock b/yarn.lock
index 5066ad9..3b41ba5 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -4,8 +4,8 @@

 "@cerc-io/laconic-sdk@0.1.4":
   version "0.1.4"
-  resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9"
-  integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ==
+  resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2"
+  integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==
   dependencies:
     "@cosmjs/amino" "^0.28.1"
     "@cosmjs/crypto" "^0.28.1"

Success. This works: ``` yarn_info_output=$(yarn info --json @cerc-io/laconic-sdk@0.1.4 2>/dev/null) package_tarball=$(echo $yarn_info_output | jq -r .data.dist.tarball) package_integrity=$(echo $yarn_info_output | jq -r .data.dist.integrity) package_shasum=$(echo $yarn_info_output | jq -r .data.dist.shasum) package_resolved=${package_tarball}#${package_shasum} escaped_package_resolved=$(printf '%s\n' "$package_resolved" | sed -e 's/[\/&]/\\&/g') if [ -n "$CERC_SCRIPT_VERBOSE" ]; then echo "Tarball: ${package_tarball}" echo "Integrity: ${package_integrity}" echo "Shasum: ${package_shasum}" echo "Resolved: ${package_resolved}" fi sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^$[[:space:]]\{1,\}resolved $.*$/\1'\"${escaped_package_resolved}\"'/' yarn.lock sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^$[[:space:]]\{1,\}integrity $.*$/\1'${package_integrity}'/' yarn.lock ``` ``` diff --git a/yarn.lock b/yarn.lock index 5066ad9..3b41ba5 100644 --- a/yarn.lock +++ b/yarn.lock @@ -4,8 +4,8 @@ "@cerc-io/laconic-sdk@0.1.4": version "0.1.4" - resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9" - integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ== + resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2" + integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw== dependencies: "@cosmjs/amino" "^0.28.1" "@cosmjs/crypto" "^0.28.1" ```

Sign in to join this conversation.

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#64