Compare commits

...
Author SHA1 Message Date
Alex e7aa2e944f Add urbit-uniswap proxy 2024-06-04 13:26:23 -07:00
Alex 7f0452cd73 Add vega stack 2024-03-29 21:50:32 -07:00
telackey d2442bcc9b revert 5308ab1e4e (#788)
revert Blind commit to fix laconic CLI calls after rename. (#784)

`laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it.

Reviewed-on: cerc-io/stack-orchestrator#784
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>

Reviewed-on: cerc-io/stack-orchestrator#788
2024-03-27 20:55:03 +00:00
nabarun 44faf36837 Update ajna-watcher-ts version for using new subgraph (#786)
Part of https://www.notion.so/Run-ajna-finance-subgraph-watcher-87748d78cd7a471b8d71f50d5fdc2657

Reviewed-on: cerc-io/stack-orchestrator#786
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
Co-committed-by: Nabarun <nabarun@deepstacksoft.com>
2024-03-26 14:52:57 +00:00
nabarun 18b006468d Update GQL path for ajna subgraph watcher server (#785)
Part of https://www.notion.so/Run-ajna-finance-subgraph-watcher-87748d78cd7a471b8d71f50d5fdc2657

Reviewed-on: cerc-io/stack-orchestrator#785
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
Co-committed-by: Nabarun <nabarun@deepstacksoft.com>
2024-03-26 11:50:05 +00:00
telackey 5308ab1e4e Blind commit to fix laconic CLI calls after rename. (#784)
`laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it.

Reviewed-on: cerc-io/stack-orchestrator#784
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-25 19:09:26 +00:00
prathamesh cd50832038 Add a Ajna watcher stack (#781)
Part of https://www.notion.so/Generate-ajna-finance-subgraph-watcher-with-codegen-5b80ac149b3f449fb138f5d92cc5485e

Reviewed-on: cerc-io/stack-orchestrator#781
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-03-21 07:17:01 +00:00
prathamesh aeddc82ebc Remove latest indexed block value from watcher alerts data (#780)
Part of https://www.notion.so/Setup-grafana-SO-stack-for-monitoring-watchers-7e23042c296c4de6b8676f1f604aa03c

Reviewed-on: cerc-io/stack-orchestrator#780
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-03-13 07:16:15 +00:00
prathamesh 17e860d6e4 Update subgraph watcher versions and instructions to use deployments (#775)
Part of https://www.notion.so/Setup-watchers-on-sandman-34b5514a10634c6fbf3ec338967c871c

Reviewed-on: cerc-io/stack-orchestrator#775
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2024-03-12 05:32:55 +00:00
telackey 523b5779be Auto-detect which certificate to use (including wildcards). (#779)
Rather than always requesting a certificate, attempt to re-use an existing certificate if it already exists in the k8s cluster.  This includes matching to a wildcard certificate.

Reviewed-on: cerc-io/stack-orchestrator#779
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-07 17:38:36 +00:00
telackey 62f7ce649d Exit non-0 if docker build fails. (#778)
Make sure to check the exit code of the docker build and bubble it back up to laconic-so.

Reviewed-on: cerc-io/stack-orchestrator#778
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-06 18:38:30 +00:00
telackey cc541ac20f Use -slim variant for Dockerfile (#773)
This saves about 1GB of space in the image.

Reviewed-on: cerc-io/stack-orchestrator#773
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-28 04:37:57 +00:00
telackey 10e2311a8b Add timed logging for the webapp build (#771)
Add lots of log and timer output to webapp builds.

Reviewed-on: cerc-io/stack-orchestrator#771
2024-02-28 00:38:11 +00:00
dboreham f32bbf9e48 Merge pull request 'Doc for fetch-containers command' (#772) from dboreham/fetch-containers-doc into main
Reviewed-on: cerc-io/stack-orchestrator#772
2024-02-27 18:45:18 +00:00
dboreham 0302153162 Doc for fetch-containers command 2024-02-27 11:44:28 -07:00
dboreham 01e4437b62 Merge pull request 'Sort order was backwards' (#770) from dboreham/fix-container-age-sort into main
Reviewed-on: cerc-io/stack-orchestrator#770
2024-02-27 16:01:17 +00:00
dboreham 64cec163b3 Sort order was backwards 2024-02-27 09:00:36 -07:00
dboreham 170ad71397 fetch-containers-fixes (#769)
Reviewed-on: cerc-io/stack-orchestrator#769
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-27 15:53:05 +00:00
dboreham da1ff609fe fetch-images command (#768)
Implementation of a command to fetch pre-built images from a remote registry, complementing the --push-images option already present on build-containers.

The two subcommands used together allow a stack to be deployed without needing to built its images, provided they have been already built and pushed to the specified container image registry.

This implementation simply picks the newest image with the right name and platform (matches against the platform Python is running on, so watch out for scenarios where Python is an x86 binary on M1 macs).

Reviewed-on: cerc-io/stack-orchestrator#768
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-27 15:15:08 +00:00
telackey 21eb9f036f Add support for pnpm as a webapp build tool. (#767)
This adds support for auto-detecting pnpm as a build tool for webapps.

Reviewed-on: cerc-io/stack-orchestrator#767
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-26 23:31:52 +00:00
telackey a0413659f7 Check for existing tag in remote repo before building. (#764)
webapps are meant to be build-once/deploy-many, but we were rebuilding them for every request.  This changes that, so that we rebuild only for every unique ApplicationRecord.

When we push the image, we now tag it according to its ApplicationRecord.

We don't want to use that tag directly in the compose file for the deployment, however, as the deployment needs to be able to adjust to new builds w/o re-writing the file all the time.  Instead, we use a per-deployment unique tag (same as before), we just update what image it references as needed.

Reviewed-on: cerc-io/stack-orchestrator#764
2024-02-24 03:22:49 +00:00
zramsayandzramsay a16fc657bf clarify uniswap urbit readme (#766)
Co-authored-by: zramsay <zach@bluecollarcoding.ca>
Reviewed-on: cerc-io/stack-orchestrator#766
2024-02-24 00:15:53 +00:00
telackey 704c42c404 Use a catchall for single page apps. (#763)
This creates a new environment variable, CERC_SINGLE_PAGE_APP, which controls whether a catchall redirection back to / is applied.

If the value is not explicitly set, we try to detect if the page looks like a single-page app.

Reviewed-on: cerc-io/stack-orchestrator#763
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-23 20:32:24 +00:00
dboreham 202f187172 Fix copy/paste error 2024-02-23 13:15:37 -07:00
dboreham aaed356d32 Simple container image publication (#762)
Reviewed-on: cerc-io/stack-orchestrator#762
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-23 19:57:47 +00:00
telackey 2af6ffce77 Tweaks for running the container registry in k8s (#760)
Minor tweaks for running the container-registry in k8s.  The big change is not requiring --image-registry.

Reviewed-on: cerc-io/stack-orchestrator#760
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
2024-02-22 21:11:06 +00:00
dboreham 7bb86cf35e Merge pull request 'Add ARM fixturenet-plugeth test job' (#759) from dboreham/arm-fixturenet-test into main
Reviewed-on: cerc-io/stack-orchestrator#759
2024-02-22 20:45:26 +00:00
dboreham 9e0892cb6b No need to start docker now 2024-02-22 13:36:00 -07:00
dboreham cf9cf6346f Add an arm-specific version of the plugeth fixturenet test 2024-02-22 13:34:37 -07:00
dboreham 642c0ead0d Add test for two config parameters (#758)
Reviewed-on: cerc-io/stack-orchestrator#758
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-22 19:35:55 +00:00
telackey 6bd77c893a Even more logging fixes (#757)
Hopefully the last one for a bit.

This only output the cmdline if log_file is present (ie, not to stdout).  It also fixes a bug where the log_file was not passed in one line.

Reviewed-on: cerc-io/stack-orchestrator#757
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-22 01:24:44 +00:00
telackey 4a4d48ddb9 Fix error when logging exception. (#756)
Reviewed-on: cerc-io/stack-orchestrator#756
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-22 00:11:06 +00:00
telackey 08438b1cd5 More logging for webapp deployments (#755)
Reviewed-on: cerc-io/stack-orchestrator#755
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-21 23:48:52 +00:00
telackey 9f1dd284a5 Better error logging for registry deployments. (#754)
We were missing errors related to registration, this should fix that.

Reviewed-on: cerc-io/stack-orchestrator#754
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-21 20:12:52 +00:00
dboreham 5985242b8c Merge pull request 'Update doc for fixturenet-laconic-loaded stack' (#753) from dboreham/update-laconic-stack-doc into main
Reviewed-on: cerc-io/stack-orchestrator#753
2024-02-21 17:33:04 +00:00
dboreham d4152b7ce3 Update doc for fixturenet-laconic-loaded stack 2024-02-21 10:29:15 -07:00
telackey db4986dcc6 snowballtool-base backend stack (#751)
This adds a stack for the backend from snowball/snowballtools-base.

Reviewed-on: cerc-io/stack-orchestrator#751
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-21 04:45:46 +00:00
telackey 65f05ea80c Run a manual build script, if present. (#750)
If the tree has a 'build-webapp.sh' script, use that.

Reviewed-on: cerc-io/stack-orchestrator#750
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-21 00:20:50 +00:00
zramsayandzramsay 01f9fe67ed add Mars v2 interface (#744)
Tested on DO with real funds on mainnet

Co-authored-by: zramsay <zach@bluecollarcoding.ca>
Reviewed-on: cerc-io/stack-orchestrator#744
2024-02-19 19:11:59 +00:00
dboreham 049ffcff71 Fix test failure 2024-02-18 12:28:48 -07:00
dboreham f5314a979b Install ed to fix CI job 2024-02-18 12:20:01 -07:00
dborehamandDavid Boreham 39f4fa4487 Container Registry Stack (#747)
Co-authored-by: David Boreham <david@bozemanpas.com>
Reviewed-on: cerc-io/stack-orchestrator#747
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-18 18:55:55 +00:00
dboreham 0b0394a940 Use absolute path for the data volume (#749)
Reviewed-on: cerc-io/stack-orchestrator#749
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-17 14:29:53 +00:00
dboreham 37b9500483 Support non-tls ingress for kind (#748)
Reviewed-on: cerc-io/stack-orchestrator#748
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-17 01:54:30 +00:00
telackey 3c3e582939 Minor envsubst improvements. (#746)
Minor fixes to envsubst for webapps.  Somewhat specially treated is `LACONIC_HOSTED_CONFIG_homepage` which can be used to replace the homepage in package.json.  With react, this gets an extra `/` though, which we need to remove.

Reviewed-on: cerc-io/stack-orchestrator#746
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-16 04:11:09 +00:00
dboreham 26d265360d Rename workflow file 2024-02-15 07:39:04 -07:00
dboreham f81b78cfbc Update .gitea/workflows/triggers/test-database 2024-02-15 14:35:56 +00:00
dboreham d9bb6b3588 Test Database Stack (#737)
Reviewed-on: cerc-io/stack-orchestrator#737
2024-02-15 05:26:29 +00:00
dboreham b59beb66eb Add simple quick deploy script (#743)
Reviewed-on: cerc-io/stack-orchestrator#743
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-15 05:00:51 +00:00
telackey 65d67dba10 Fix k8s and enable it by default on PRs (#742)
Reviewed-on: cerc-io/stack-orchestrator#742
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-14 23:50:09 +00:00
telackey b22c72e715 For k8s, use provisioner-managed volumes when an absolute host path is not specified. (#741)
In kind, when we bind-mount a host directory it is first mounted into the kind container at /mnt, then into the pod at the desired location.

We accidentally picked this up for full-blown k8s, and were creating volumes at /mnt.  This changes the behavior for both kind and regular k8s so that bind mounts are only allowed if a fully-qualified path is specified.  If no path is specified at all, a default storageClass is assumed to be present, and the volume managed by a provisioner.

Eg, for kind, the default provisioner is: https://github.com/rancher/local-path-provisioner

```
stack: test
deploy-to: k8s-kind
config:
  test-variable-1: test-value-1
network:
  ports:
    test:
     - '80'
volumes:
  # this will be bind-mounted to a host-path
  test-data-bind: /srv/data
  # this will be managed by the k8s node
  test-data-auto:
configmaps:
  test-config: ./configmap/test-config
```

Reviewed-on: cerc-io/stack-orchestrator#741
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-14 21:45:01 +00:00
telackey c9444591f5 Fix default webapp port number. (#740)
Reviewed-on: cerc-io/stack-orchestrator#740
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-09 01:20:41 +00:00
telackey 903f3b10e2 Add support for annotations and labels in spec. (#739)
```
stack: webapp-deployer-backend
deploy-to: k8s
annotations:
  foo.bar.annot/{name}: baz
labels:
  a.b.c/{name}.blah: "value"
```

Reviewed-on: cerc-io/stack-orchestrator#739
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-09 00:11:07 +00:00
telackey 72ed2eb91a Fix bad test in tag check. (#738)
Reviewed-on: cerc-io/stack-orchestrator#738
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-08 20:38:41 +00:00
dboreham 2104eb5f30 Merge pull request 'Add Mars stack' (#725) from zach/mars into main
Reviewed-on: cerc-io/stack-orchestrator#725
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
2024-02-08 20:30:47 +00:00
afd6be3b13 Ping pub (#663)
for #170, revives #190
uses https://github.com/LaconicNetwork/explorer/pull/1

Co-authored-by: zramsay <zach@bluecollarcoding.ca>
Co-authored-by: David Boreham <david@bozemanpass.com>
Reviewed-on: cerc-io/stack-orchestrator#663
Co-authored-by: zramsay <zramsay@noreply.git.vdb.to>
Co-committed-by: zramsay <zramsay@noreply.git.vdb.to>
2024-02-08 20:13:12 +00:00
dboreham f914baa913 Merge branch 'main' into zach/mars 2024-02-08 19:52:49 +00:00
dboreham 8be1e684e8 Process environment variables defined in compose files (#736)
Reviewed-on: cerc-io/stack-orchestrator#736
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-08 19:41:57 +00:00
telackey 5d16251ce9 Merge pull request 'Add resource limit options to spec.' (#735) from telackey/limits into main
Reviewed-on: cerc-io/stack-orchestrator#735
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
2024-02-08 16:52:49 +00:00
telackey 3309782439 Refactor 2024-02-08 00:47:46 -06:00
telackey 4b3b3478e7 Switch to Docker-style limits 2024-02-08 00:43:41 -06:00
telackey 2a9955055c debug 2024-02-07 16:56:35 -06:00
telackey 8964e1c0fe Add resource limit options to spec. 2024-02-07 16:48:02 -06:00
telackey d2ebb81d77 Tags for undeploy (#734)
```
  --include-tags TEXT             Only include requests with matching tags
                                  (comma-separated).
  --exclude-tags TEXT             Exclude requests with matching tags (comma-
                                  separated).
```

Reviewed-on: cerc-io/stack-orchestrator#734
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-07 21:45:16 +00:00
telackey 4a981d8d2e Fix repo URL (#733)
Needs a '/' (http) not ':' (ssh).

Reviewed-on: cerc-io/stack-orchestrator#733
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-07 18:52:07 +00:00
telackey 88a0236ca9 Add the ability to filter deployment requests by tag. (#730)
Reviewed-on: cerc-io/stack-orchestrator#730
2024-02-07 03:12:40 +00:00
telackey 937b983ec9 Update links from github.com to git.vdb.to (#732)
Update links and references to github.com to git.vdb.to.

Also enable the flake8 lint action in gitea.

Reviewed-on: cerc-io/stack-orchestrator#732
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-06 22:42:44 +00:00
dboreham bfbcfb7904 Volume processing fixes (#729) 2024-02-06 12:32:10 -07:00
dboreham 3d5ececba5 Remove duplicate plugin paths and resulting extraneous error message (#728) 2024-02-06 11:59:37 -07:00
dboreham 6848fc33cf Implement dry run support for k8s deploy (#727) 2024-02-06 07:07:56 -07:00
telackeyanddboreham 36bb068983 Add ConfigMap test. (#726)
* Add ConfigMap test.

* eof

* Minor tweak

* Trigger test

---------

Co-authored-by: David Boreham <david@bozemanpass.com>
2024-02-05 14:15:11 -06:00
dboreham 25a2b70f2c Fix command in mainnet-eth docs 2024-02-03 18:25:02 -07:00
telackey 2fcd416e29 Basic webapp deployer stack. (#722) 2024-02-02 19:05:15 -07:00
Zach 098567625a Create README.md 2024-01-30 17:47:56 -05:00
zramsay 23ee3e19b7 mars: add env vars to docker-compose 2024-01-29 22:44:55 +00:00
zramsay 2d764fc7d0 basic mars stack 2024-01-29 16:00:58 +00:00
168 changed files with 6138 additions and 878 deletions
@@ -0,0 +1,45 @@
name: Fixturenet-Eth-Plugeth-Arm-Test
on:
push:
branches: '*'
paths:
- '!**'
- '.gitea/workflows/triggers/fixturenet-eth-plugeth-arm-test'
schedule: # Note: coordinate with other tests to not overload runners at the same time of day
- cron: '2 14 * * *'
# Needed until we can incorporate docker startup into the executor container
env:
DOCKER_HOST: unix:///var/run/dind.sock
jobs:
test:
name: "Run an Ethereum plugeth fixturenet test"
runs-on: ubuntu-latest-arm
steps:
- name: "Clone project repository"
uses: actions/checkout@v3
# At present the stock setup-python action fails on Linux/aarch64
# Conditional steps below workaroud this by using deadsnakes for that case only
- name: "Install Python for ARM on Linux"
if: ${{ runner.arch == 'arm64' && runner.os == 'Linux' }}
uses: deadsnakes/action@v3.0.1
with:
python-version: '3.8'
- name: "Install Python cases other than ARM on Linux"
if: ${{ ! (runner.arch == 'arm64' && runner.os == 'Linux') }}
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: "Print Python version"
run: python3 --version
- name: "Install shiv"
run: pip install shiv
- name: "Generate build version file"
run: ./scripts/create_build_tag_file.sh
- name: "Build local shiv package"
run: ./scripts/build_shiv_package.sh
- name: "Run fixturenet-eth tests"
run: ./tests/fixturenet-eth-plugeth/run-test.sh
+21
View File
@@ -0,0 +1,21 @@
name: Lint Checks
on:
pull_request:
branches: '*'
push:
branches: '*'
jobs:
test:
name: "Run linter"
runs-on: ubuntu-latest
steps:
- name: "Clone project repository"
uses: actions/checkout@v3
- name: "Install Python"
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name : "Run flake8"
uses: py-actions/flake8@v2
@@ -0,0 +1,54 @@
name: Container Registry Test
on:
push:
branches: '*'
paths:
- '!**'
- '.gitea/workflows/triggers/test-container-registry'
- '.gitea/workflows/test-container-registry.yml'
- 'tests/container-registry/run-test.sh'
schedule: # Note: coordinate with other tests to not overload runners at the same time of day
- cron: '6 19 * * *'
jobs:
test:
name: "Run contaier registry hosting test on kind/k8s"
runs-on: ubuntu-22.04
steps:
- name: "Clone project repository"
uses: actions/checkout@v3
# At present the stock setup-python action fails on Linux/aarch64
# Conditional steps below workaroud this by using deadsnakes for that case only
- name: "Install Python for ARM on Linux"
if: ${{ runner.arch == 'arm64' && runner.os == 'Linux' }}
uses: deadsnakes/action@v3.0.1
with:
python-version: '3.8'
- name: "Install Python cases other than ARM on Linux"
if: ${{ ! (runner.arch == 'arm64' && runner.os == 'Linux') }}
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: "Print Python version"
run: python3 --version
- name: "Install shiv"
run: pip install shiv
- name: "Generate build version file"
run: ./scripts/create_build_tag_file.sh
- name: "Build local shiv package"
run: ./scripts/build_shiv_package.sh
- name: "Check cgroups version"
run: mount | grep cgroup
- name: "Install kind"
run: ./tests/scripts/install-kind.sh
- name: "Install Kubectl"
run: ./tests/scripts/install-kubectl.sh
- name: "Install ed" # Only needed until we remove the need to edit the spec file
run: apt update && apt install -y ed
- name: "Run container registry deployment test"
run: |
source /opt/bash-utils/cgroup-helper.sh
join_cgroup
./tests/container-registry/run-test.sh
+52
View File
@@ -0,0 +1,52 @@
name: Database Test
on:
push:
branches: '*'
paths:
- '!**'
- '.gitea/workflows/triggers/test-database'
- '.gitea/workflows/test-database.yml'
- 'tests/database/run-test.sh'
schedule: # Note: coordinate with other tests to not overload runners at the same time of day
- cron: '5 18 * * *'
jobs:
test:
name: "Run database hosting test on kind/k8s"
runs-on: ubuntu-22.04
steps:
- name: "Clone project repository"
uses: actions/checkout@v3
# At present the stock setup-python action fails on Linux/aarch64
# Conditional steps below workaroud this by using deadsnakes for that case only
- name: "Install Python for ARM on Linux"
if: ${{ runner.arch == 'arm64' && runner.os == 'Linux' }}
uses: deadsnakes/action@v3.0.1
with:
python-version: '3.8'
- name: "Install Python cases other than ARM on Linux"
if: ${{ ! (runner.arch == 'arm64' && runner.os == 'Linux') }}
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: "Print Python version"
run: python3 --version
- name: "Install shiv"
run: pip install shiv
- name: "Generate build version file"
run: ./scripts/create_build_tag_file.sh
- name: "Build local shiv package"
run: ./scripts/build_shiv_package.sh
- name: "Check cgroups version"
run: mount | grep cgroup
- name: "Install kind"
run: ./tests/scripts/install-kind.sh
- name: "Install Kubectl"
run: ./tests/scripts/install-kubectl.sh
- name: "Run database deployment test"
run: |
source /opt/bash-utils/cgroup-helper.sh
join_cgroup
./tests/database/run-test.sh
+2
View File
@@ -1,6 +1,8 @@
name: K8s Deploy Test
on:
pull_request:
branches: '*'
push:
branches: '*'
paths:
@@ -0,0 +1,2 @@
Change this file to trigger running the fixturenet-eth-plugeth-arm-test CI job
@@ -0,0 +1 @@
Change this file to trigger running the test-container-registry CI job
+2
View File
@@ -0,0 +1,2 @@
Change this file to trigger running the test-database CI job
Trigger test run
@@ -1 +1,2 @@
Change this file to trigger running the test-k8s-deploy CI job
Trigger test on PR branch
+3 -3
View File
@@ -29,10 +29,10 @@ chmod +x ~/.docker/cli-plugins/docker-compose
Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be
a "user" binary directory such as `~/bin` or perhaps `/usr/local/laconic` or possibly just the current working directory.
Now, having selected that directory, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
Now, having selected that directory, download the latest release from [this page](https://git.vdb.to/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
```bash
curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
curl -L -o ~/bin/laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
```
Give it execute permissions:
@@ -52,7 +52,7 @@ Version: 1.1.0-7a607c2-202304260513
Save the distribution url to `~/.laconic-so/config.yml`:
```bash
mkdir ~/.laconic-so
echo "distribution-url: https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so" > ~/.laconic-so/config.yml
echo "distribution-url: https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so" > ~/.laconic-so/config.yml
```
### Update
+1 -1
View File
@@ -26,7 +26,7 @@ In addition to the pre-requisites listed in the [README](/README.md), the follow
1. Clone this repository:
```
$ git clone https://github.com/cerc-io/stack-orchestrator.git
$ git clone https://git.vdb.to/cerc-io/stack-orchestrator.git
```
2. Enter the project directory:
+2 -2
View File
@@ -1,10 +1,10 @@
# Adding a new stack
See [this PR](https://github.com/cerc-io/stack-orchestrator/pull/434) for an example of how to currently add a minimal stack to stack orchestrator. The [reth stack](https://github.com/cerc-io/stack-orchestrator/pull/435) is another good example.
See [this PR](https://git.vdb.to/cerc-io/stack-orchestrator/pull/434) for an example of how to currently add a minimal stack to stack orchestrator. The [reth stack](https://git.vdb.to/cerc-io/stack-orchestrator/pull/435) is another good example.
For external developers, we recommend forking this repo and adding your stack directly to your fork. This initially requires running in "developer mode" as described [here](/docs/CONTRIBUTING.md). Check out the [Namada stack](https://github.com/vknowable/stack-orchestrator/blob/main/app/data/stacks/public-namada/digitalocean_quickstart.md) from Knowable to see how that is done.
Core to the feature completeness of stack orchestrator is to [decouple the tool functionality from payload](https://github.com/cerc-io/stack-orchestrator/issues/315) which will no longer require forking to add a stack.
Core to the feature completeness of stack orchestrator is to [decouple the tool functionality from payload](https://git.vdb.to/cerc-io/stack-orchestrator/issues/315) which will no longer require forking to add a stack.
## Example
+9
View File
@@ -0,0 +1,9 @@
# Fetching pre-built container images
When Stack Orchestrator deploys a stack containing a suite of one or more containers it expects images for those containers to be on the local machine with a tag of the form `<image-name>:local` Images for these containers can be built from source (and optionally base container images from public registries) with the `build-containers` subcommand.
However, the task of building a large number of containers from source may consume considerable time and machine resources. This is where the `fetch-containers` subcommand steps in. It is designed to work exactly like `build-containers` but instead the images, pre-built, are fetched from an image registry then re-tagged for deployment. It can be used in place of `build-containers` for any stack provided the necessary containers, built for the local machine architecture (e.g. arm64 or x86-64) have already been published in an image registry.
## Usage
To use `fetch-containers`, provide an image registry path, a username and token/password with read access to the registry, and optionally specify `--force-local-overwrite`. If this argument is not specified, if there is already a locally built or previously fetched image for a stack container on the machine, it will not be overwritten and a warning issued.
```
$ laconic-so --stack mobymask-v3-demo fetch-containers --image-registry git.vdb.to/cerc-io --registry-username <registry-user> --registry-token <registry-token> --force-local-overwrite
```
+104 -97
View File
@@ -15,132 +15,139 @@ To avoid hiccups on Mac M1/M2 and any local machine nuances that may affect the
16 GB Memory / 8 Intel vCPUs / 160 GB Disk.
1. Login to the droplet as root (either by SSH key or password set in the DO console)
```
ssh root@IP
```
```
ssh root@IP
```
1. Get the install script, give it executable permissions, and run it:
2. Get the install script, give it executable permissions, and run it:
```
curl -o install.sh https://raw.githubusercontent.com/cerc-io/stack-orchestrator/main/scripts/quick-install-linux.sh
```
```
chmod +x install.sh
```
```
bash install.sh
```
```
curl -o install.sh https://raw.githubusercontent.com/cerc-io/stack-orchestrator/main/scripts/quick-install-linux.sh
```
```
chmod +x install.sh
```
```
bash install.sh
```
1. Confirm docker was installed and activate the changes in `~/.profile`:
3. Confirm docker was installed and activate the changes in `~/.profile`:
```
docker run hello-world
```
```
source ~/.profile
```
```
docker run hello-world
```
```
source ~/.profile
```
1. Verify installation:
4. Verify installation:
```
laconic-so version
```
```
laconic-so version
```
## Setup the laconic fixturenet stack
1. Get the repositories
```
laconic-so --stack fixturenet-laconic-loaded setup-repositories --include git.vdb.to/cerc-io/laconicd,git.vdb.to/cerc-io/laconic-sdk,git.vdb.to/cerc-io/laconic-registry-cli,git.vdb.to/cerc-io/laconic-console
```
```
laconic-so --stack fixturenet-laconic-loaded setup-repositories --include git.vdb.to/cerc-io/laconicd,git.vdb.to/cerc-io/laconic-sdk,git.vdb.to/cerc-io/laconic-registry-cli,git.vdb.to/cerc-io/laconic-console
```
2. Set this environment variable to the Laconic self-hosted Gitea instance:
1. Build the containers:
```
export CERC_NPM_REGISTRY_URL=https://git.vdb.to/api/packages/cerc-io/npm/
```
```
laconic-so --stack fixturenet-laconic-loaded build-containers
```
3. Build the containers:
It's possible to run into an `ESOCKETTIMEDOUT` error, e.g., `error An unexpected error occurred: "https://registry.yarnpkg.com/@material-ui/icons/-/icons-4.11.3.tgz: ESOCKETTIMEDOUT"`. This may happen even if you have a great internet connection. In that case, re-run the `build-containers` command.
```
laconic-so --stack fixturenet-laconic-loaded build-containers
```
It's possible to run into an `ESOCKETTIMEDOUT` error, e.g., `error An unexpected error occurred: "https://registry.yarnpkg.com/@material-ui/icons/-/icons-4.11.3.tgz: ESOCKETTIMEDOUT"`. This may happen even if you have a great internet connection. In that case, re-run the `build-containers` command.
1. Set this environment variable to your droplet's IP address or fully qualified DNS host name if it has one:
4. Set this environment variable to your droplet's IP address:
```
export BACKEND_ENDPOINT=http://<your-IP-or-hostname>:9473
```
e.g.
```
export BACKEND_ENDPOINT=http://my-test-server.example.com:9473
```
```
export LACONIC_HOSTED_ENDPOINT=http://<your-IP>
```
1. Create a deployment directory for the stack:
```
laconic-so --stack fixturenet-laconic-loaded deploy init --output laconic-loaded.spec --map-ports-to-host any-same --config LACONIC_HOSTED_ENDPOINT=$BACKEND_ENDPOINT
```
```
laconic-so --stack fixturenet-laconic-loaded deploy create --deployment-dir laconic-loaded-deployment --spec-file laconic-loaded.spec
```
2. Start the stack:
5. Deploy the stack:
```
laconic-so deployment --dir laconic-loaded-deployment start
```
```
laconic-so --stack fixturenet-laconic-loaded deploy up
```
3. Check the logs:
6. Check the logs:
```
laconic-so deployment --dir laconic-loaded-deployment logs
```
```
laconic-so --stack fixturenet-laconic-loaded deploy logs
```
You'll see output from `laconicd` and the block height should be >1 to confirm it is running:
You'll see output from `laconicd` and the block height should be >1 to confirm it is running:
```
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:29PM INF indexed block exents height=12 module=txindex server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF Timed out dur=4976.960115 height=13 module=consensus round=0 server=node step=1
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4","parts":{"hash":"22411A20B7F14CDA33244420FBDDAF24450C0628C7A06034FF22DAC3699DDCC8","total":1}},"height":13,"pol_round":-1,"round":0,"signature":"DEuqnaQmvyYbUwckttJmgKdpRu6eVm9i+9rQ1pIrV2PidkMNdWRZBLdmNghkIrUzGbW8Xd7UVJxtLRmwRASgBg==","timestamp":"2023-04-18T21:30:01.49450663Z"} server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received complete proposal block hash=D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4 height=13 module=consensus server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF finalizing commit of block hash={} height=13 module=consensus num_txs=0 root=1A8CA1AF139CCC80EC007C6321D8A63A46A793386EE2EDF9A5CA0AB2C90728B7 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF minted coins from module account amount=2059730459416582643aphoton from=mint module=x/bank
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF executed block height=13 module=state num_invalid_txs=0 num_valid_txs=0 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF commit synced commit=436F6D6D697449447B5B363520313037203630203232372039352038352032303820313334203231392032303520313433203130372031343920313431203139203139322038362031323720362031383520323533203137362031333820313735203135392031383620323334203135382031323120313431203230342037335D3A447D
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF committed state app_hash=416B3CE35F55D086DBCD8F6B958D13C0567F06B9FDB08AAF9FBAEA9E798DCC49 height=13 module=state num_txs=0 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF indexed block exents height=13 module=txindex server=node
```
```
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:29PM INF indexed block exents height=12 module=txindex server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF Timed out dur=4976.960115 height=13 module=consensus round=0 server=node step=1
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4","parts":{"hash":"22411A20B7F14CDA33244420FBDDAF24450C0628C7A06034FF22DAC3699DDCC8","total":1}},"height":13,"pol_round":-1,"round":0,"signature":"DEuqnaQmvyYbUwckttJmgKdpRu6eVm9i+9rQ1pIrV2PidkMNdWRZBLdmNghkIrUzGbW8Xd7UVJxtLRmwRASgBg==","timestamp":"2023-04-18T21:30:01.49450663Z"} server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received complete proposal block hash=D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4 height=13 module=consensus server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF finalizing commit of block hash={} height=13 module=consensus num_txs=0 root=1A8CA1AF139CCC80EC007C6321D8A63A46A793386EE2EDF9A5CA0AB2C90728B7 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF minted coins from module account amount=2059730459416582643aphoton from=mint module=x/bank
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF executed block height=13 module=state num_invalid_txs=0 num_valid_txs=0 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF commit synced commit=436F6D6D697449447B5B363520313037203630203232372039352038352032303820313334203231392032303520313433203130372031343920313431203139203139322038362031323720362031383520323533203137362031333820313735203135392031383620323334203135382031323120313431203230342037335D3A447D
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF committed state app_hash=416B3CE35F55D086DBCD8F6B958D13C0567F06B9FDB08AAF9FBAEA9E798DCC49 height=13 module=state num_txs=0 server=node
laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF indexed block exents height=13 module=txindex server=node
```
4. Confirm operation of the registry CLI:
7. Confirm operation of the registry CLI:
```
laconic-so deployment --dir laconic-loaded-deployment exec cli "laconic cns status"
```
```
laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns status"
```
```
{
"version": "0.3.0",
"node": {
"id": "4216af2ac9f68bda33a38803fc1b5c9559312c1d",
"network": "laconic_9000-1",
"moniker": "localtestnet"
},
"sync": {
"latest_block_hash": "1BDF4CB9AE2390DA65BCF997C83133C18014FCDDCAE03708488F0B56FCEEA429",
"latest_block_height": "5",
"latest_block_time": "2023-08-09 16:00:30.386903172 +0000 UTC",
"catching_up": false
},
"validator": {
"address": "651FBC700B747C76E90ACFC18CC9508C3D0905B9",
"voting_power": "1000000000000000"
},
"validators": [
{
"address": "651FBC700B747C76E90ACFC18CC9508C3D0905B9",
"voting_power": "1000000000000000",
"proposer_priority": "0"
}
],
"num_peers": "0",
"peers": [],
"disk_usage": "292.0K"
}
```
```
{
"version": "0.3.0",
"node": {
"id": "4216af2ac9f68bda33a38803fc1b5c9559312c1d",
"network": "laconic_9000-1",
"moniker": "localtestnet"
},
"sync": {
"latest_block_hash": "1BDF4CB9AE2390DA65BCF997C83133C18014FCDDCAE03708488F0B56FCEEA429",
"latest_block_height": "5",
"latest_block_time": "2023-08-09 16:00:30.386903172 +0000 UTC",
"catching_up": false
},
"validator": {
"address": "651FBC700B747C76E90ACFC18CC9508C3D0905B9",
"voting_power": "1000000000000000"
},
"validators": [
{
"address": "651FBC700B747C76E90ACFC18CC9508C3D0905B9",
"voting_power": "1000000000000000",
"proposer_priority": "0"
}
],
"num_peers": "0",
"peers": [],
"disk_usage": "292.0K"
}
```
## Configure Digital Ocean firewall
(Note this step may not be necessary depending on the droplet image used)
Let's open some ports.
1. In the Digital Ocean web console, navigate to your droplet's main page. Select the "Networking" tab and scroll down to "Firewall".
+1 -1
View File
@@ -1,6 +1,6 @@
# Specification
Note: this page is out of date (but still useful) - it will no longer be useful once stacks are [decoupled from the tool functionality](https://github.com/cerc-io/stack-orchestrator/issues/315).
Note: this page is out of date (but still useful) - it will no longer be useful once stacks are [decoupled from the tool functionality](https://git.vdb.to/cerc-io/stack-orchestrator/issues/315).
## Implementation
+1
View File
@@ -10,3 +10,4 @@ pydantic==1.10.9
tomli==2.0.1
validators==0.22.0
kubernetes>=28.1.0
humanfriendly>=10.0
+1 -1
View File
@@ -41,4 +41,4 @@ runcmd:
- apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
- systemctl enable docker
- systemctl start docker
- git clone https://github.com/cerc-io/stack-orchestrator.git /home/ubuntu/stack-orchestrator
- git clone https://git.vdb.to/cerc-io/stack-orchestrator.git /home/ubuntu/stack-orchestrator
+1 -1
View File
@@ -31,5 +31,5 @@ runcmd:
- apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
- systemctl enable docker
- systemctl start docker
- curl -L -o /usr/local/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
- curl -L -o /usr/local/bin/laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
- chmod +x /usr/local/bin/laconic-so
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Beginnings of a script to quickly spin up and test a deployment
if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then
set -x
fi
if [[ -n "$1" ]]; then
stack_name=$1
else
stack_name="test"
fi
spec_file_name="${stack_name}-spec.yml"
deployment_dir_name="${stack_name}-deployment"
rm -f ${spec_file_name}
rm -rf ${deployment_dir_name}
laconic-so --stack ${stack_name} deploy --deploy-to compose init --output ${spec_file_name}
laconic-so --stack ${stack_name} deploy --deploy-to compose create --deployment-dir ${deployment_dir_name} --spec-file ${spec_file_name}
#laconic-so deployment --dir ${deployment_dir_name} start
#laconic-so deployment --dir ${deployment_dir_name} ps
#laconic-so deployment --dir ${deployment_dir_name} stop
+1 -1
View File
@@ -137,7 +137,7 @@ fi
echo "**************************************************************************************"
echo "Installing laconic-so"
# install latest `laconic-so`
distribution_url=https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
distribution_url=https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
install_filename=${install_dir}/laconic-so
mkdir -p ${install_dir}
curl -L -o ${install_filename} ${distribution_url}
+1 -1
View File
@@ -13,7 +13,7 @@ setup(
description='Orchestrates deployment of the Laconic stack',
long_description=long_description,
long_description_content_type="text/markdown",
url='https://github.com/cerc-io/stack-orchestrator',
url='https://git.vdb.to/cerc-io/stack-orchestrator',
py_modules=['stack_orchestrator'],
packages=find_packages(),
install_requires=[requirements],
+61 -66
View File
@@ -25,10 +25,13 @@ import sys
from decouple import config
import subprocess
import click
import importlib.resources
from pathlib import Path
from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config, stack_is_external
from stack_orchestrator.opts import opts
from stack_orchestrator.util import include_exclude_check, stack_is_external, error_exit
from stack_orchestrator.base import get_npm_registry_url
from stack_orchestrator.build.build_types import BuildContext
from stack_orchestrator.build.publish import publish_image
from stack_orchestrator.build.build_util import get_containers_in_scope
# TODO: find a place for this
# epilog="Config provided either in .env or settings.ini or env vars: CERC_REPO_BASE_DIR (defaults to ~/cerc)"
@@ -59,69 +62,58 @@ def make_container_build_env(dev_root_path: str,
return container_build_env
def process_container(stack: str,
container,
container_build_dir: str,
container_build_env: dict,
dev_root_path: str,
quiet: bool,
verbose: bool,
dry_run: bool,
continue_on_error: bool,
):
if not quiet:
print(f"Building: {container}")
def process_container(build_context: BuildContext) -> bool:
if not opts.o.quiet:
print(f"Building: {build_context.container}")
default_container_tag = f"{container}:local"
container_build_env.update({"CERC_DEFAULT_CONTAINER_IMAGE_TAG": default_container_tag})
default_container_tag = f"{build_context.container}:local"
build_context.container_build_env.update({"CERC_DEFAULT_CONTAINER_IMAGE_TAG": default_container_tag})
# Check if this is in an external stack
if stack_is_external(stack):
container_parent_dir = Path(stack).joinpath("container-build")
temp_build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
if stack_is_external(build_context.stack):
container_parent_dir = Path(build_context.stack).joinpath("container-build")
temp_build_dir = container_parent_dir.joinpath(build_context.container.replace("/", "-"))
temp_build_script_filename = temp_build_dir.joinpath("build.sh")
# Now check if the container exists in the external stack.
if not temp_build_script_filename.exists():
# If not, revert to building an internal container
container_parent_dir = container_build_dir
container_parent_dir = build_context.container_build_dir
else:
container_parent_dir = container_build_dir
container_parent_dir = build_context.container_build_dir
build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
build_dir = container_parent_dir.joinpath(build_context.container.replace("/", "-"))
build_script_filename = build_dir.joinpath("build.sh")
if verbose:
if opts.o.verbose:
print(f"Build script filename: {build_script_filename}")
if os.path.exists(build_script_filename):
build_command = build_script_filename.as_posix()
else:
if verbose:
if opts.o.verbose:
print(f"No script file found: {build_script_filename}, using default build script")
repo_dir = container.split('/')[1]
repo_dir = build_context.container.split('/')[1]
# TODO: make this less of a hack -- should be specified in some metadata somewhere
# Check if we have a repo for this container. If not, set the context dir to the container-build subdir
repo_full_path = os.path.join(dev_root_path, repo_dir)
repo_full_path = os.path.join(build_context.dev_root_path, repo_dir)
repo_dir_or_build_dir = repo_full_path if os.path.exists(repo_full_path) else build_dir
build_command = os.path.join(container_build_dir,
build_command = os.path.join(build_context.container_build_dir,
"default-build.sh") + f" {default_container_tag} {repo_dir_or_build_dir}"
if not dry_run:
if not opts.o.dry_run:
# No PATH at all causes failures with podman.
if "PATH" not in container_build_env:
container_build_env["PATH"] = os.environ["PATH"]
if verbose:
print(f"Executing: {build_command} with environment: {container_build_env}")
build_result = subprocess.run(build_command, shell=True, env=container_build_env)
if verbose:
if "PATH" not in build_context.container_build_env:
build_context.container_build_env["PATH"] = os.environ["PATH"]
if opts.o.verbose:
print(f"Executing: {build_command} with environment: {build_context.container_build_env}")
build_result = subprocess.run(build_command, shell=True, env=build_context.container_build_env)
if opts.o.verbose:
print(f"Return code is: {build_result.returncode}")
if build_result.returncode != 0:
print(f"Error running build for {container}")
if not continue_on_error:
print("FATAL Error: container build failed and --continue-on-error not set, exiting")
sys.exit(1)
else:
print("****** Container Build Error, continuing because --continue-on-error is set")
return False
else:
return True
else:
print("Skipped")
return True
@click.command()
@@ -129,17 +121,14 @@ def process_container(stack: str,
@click.option('--exclude', help="don\'t build these containers")
@click.option("--force-rebuild", is_flag=True, default=False, help="Override dependency checking -- always rebuild")
@click.option("--extra-build-args", help="Supply extra arguments to build")
@click.option("--publish-images", is_flag=True, default=False, help="Publish the built images in the specified image registry")
@click.option("--image-registry", help="Specify the image registry for --publish-images")
@click.pass_context
def command(ctx, include, exclude, force_rebuild, extra_build_args):
def command(ctx, include, exclude, force_rebuild, extra_build_args, publish_images, image_registry):
'''build the set of containers required for a complete stack'''
quiet = ctx.obj.quiet
verbose = ctx.obj.verbose
dry_run = ctx.obj.dry_run
debug = ctx.obj.debug
local_stack = ctx.obj.local_stack
stack = ctx.obj.stack
continue_on_error = ctx.obj.continue_on_error
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
container_build_dir = Path(__file__).absolute().parent.parent.joinpath("data", "container-build")
@@ -150,39 +139,45 @@ def command(ctx, include, exclude, force_rebuild, extra_build_args):
else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
if not quiet:
if not opts.o.quiet:
print(f'Dev Root is: {dev_root_path}')
if not os.path.isdir(dev_root_path):
print('Dev root directory doesn\'t exist, creating')
# See: https://stackoverflow.com/a/20885799/1701505
from stack_orchestrator import data
with importlib.resources.open_text(data, "container-image-list.txt") as container_list_file:
all_containers = container_list_file.read().splitlines()
if publish_images:
if not image_registry:
error_exit("--image-registry must be supplied with --publish-images")
containers_in_scope = []
if stack:
stack_config = get_parsed_stack_config(stack)
containers_in_scope = stack_config['containers']
else:
containers_in_scope = all_containers
if verbose:
print(f'Containers: {containers_in_scope}')
if stack:
print(f"Stack: {stack}")
containers_in_scope = get_containers_in_scope(stack)
container_build_env = make_container_build_env(dev_root_path,
container_build_dir,
debug,
opts.o.debug,
force_rebuild,
extra_build_args)
for container in containers_in_scope:
if include_exclude_check(container, include, exclude):
process_container(stack, container, container_build_dir, container_build_env,
dev_root_path, quiet, verbose, dry_run, continue_on_error)
build_context = BuildContext(
stack,
container,
container_build_dir,
container_build_env,
dev_root_path
)
result = process_container(build_context)
if result:
if publish_images:
publish_image(f"{container}:local", image_registry)
else:
print(f"Error running build for {build_context.container}")
if not opts.o.continue_on_error:
error_exit("container build failed and --continue-on-error not set, exiting")
sys.exit(1)
else:
print("****** Container Build Error, continuing because --continue-on-error is set")
else:
if verbose:
if opts.o.verbose:
print(f"Excluding: {container}")
+29
View File
@@ -0,0 +1,29 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
@dataclass
class BuildContext:
stack: str
container: str
container_build_dir: Path
container_build_env: Mapping[str,str]
dev_root_path: str
+43
View File
@@ -0,0 +1,43 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import importlib.resources
from stack_orchestrator.opts import opts
from stack_orchestrator.util import get_parsed_stack_config, warn_exit
def get_containers_in_scope(stack: str):
# See: https://stackoverflow.com/a/20885799/1701505
from stack_orchestrator import data
with importlib.resources.open_text(data, "container-image-list.txt") as container_list_file:
all_containers = container_list_file.read().splitlines()
containers_in_scope = []
if stack:
stack_config = get_parsed_stack_config(stack)
if "containers" not in stack_config or stack_config["containers"] is None:
warn_exit(f"stack {stack} does not define any containers")
containers_in_scope = stack_config['containers']
else:
containers_in_scope = all_containers
if opts.o.verbose:
print(f'Containers: {containers_in_scope}')
if stack:
print(f"Stack: {stack}")
return containers_in_scope
+47 -14
View File
@@ -21,11 +21,14 @@
# TODO: display the available list of containers; allow re-build of either all or specific containers
import os
import sys
from decouple import config
import click
from pathlib import Path
from stack_orchestrator.build import build_containers
from stack_orchestrator.deploy.webapp.util import determine_base_container
from stack_orchestrator.deploy.webapp.util import determine_base_container, TimedLogger
from stack_orchestrator.build.build_types import BuildContext
@click.command()
@@ -37,26 +40,25 @@ from stack_orchestrator.deploy.webapp.util import determine_base_container
@click.pass_context
def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, tag):
'''build the specified webapp container'''
logger = TimedLogger()
quiet = ctx.obj.quiet
verbose = ctx.obj.verbose
dry_run = ctx.obj.dry_run
debug = ctx.obj.debug
verbose = ctx.obj.verbose
local_stack = ctx.obj.local_stack
stack = ctx.obj.stack
continue_on_error = ctx.obj.continue_on_error
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
container_build_dir = Path(__file__).absolute().parent.parent.joinpath("data", "container-build")
if local_stack:
dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
logger.log(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
if not quiet:
print(f'Dev Root is: {dev_root_path}')
if verbose:
logger.log(f'Dev Root is: {dev_root_path}')
if not base_container:
base_container = determine_base_container(source_repo)
@@ -65,8 +67,23 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
container_build_env = build_containers.make_container_build_env(dev_root_path, container_build_dir, debug,
force_rebuild, extra_build_args)
build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
verbose, dry_run, continue_on_error)
if verbose:
logger.log(f"Building base container: {base_container}")
build_context_1 = BuildContext(
stack,
base_container,
container_build_dir,
container_build_env,
dev_root_path,
)
ok = build_containers.process_container(build_context_1)
if not ok:
logger.log("ERROR: Build failed.")
sys.exit(1)
if verbose:
logger.log(f"Base container {base_container} build finished.")
# Now build the target webapp. We use the same build script, but with a different Dockerfile and work dir.
container_build_env["CERC_WEBAPP_BUILD_RUNNING"] = "true"
@@ -76,9 +93,25 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
"Dockerfile.webapp")
if not tag:
webapp_name = os.path.abspath(source_repo).split(os.path.sep)[-1]
container_build_env["CERC_CONTAINER_BUILD_TAG"] = f"cerc/{webapp_name}:local"
else:
container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
tag = f"cerc/{webapp_name}:local"
build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
verbose, dry_run, continue_on_error)
container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
if verbose:
logger.log(f"Building app container: {tag}")
build_context_2 = BuildContext(
stack,
base_container,
container_build_dir,
container_build_env,
dev_root_path,
)
ok = build_containers.process_container(build_context_2)
if not ok:
logger.log("ERROR: Build failed.")
sys.exit(1)
if verbose:
logger.log(f"App container {base_container} build finished.")
logger.log("build-webapp complete", show_step_time=False, show_total_time=True)
@@ -0,0 +1,195 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import click
from dataclasses import dataclass
import json
import platform
from python_on_whales import DockerClient
from python_on_whales.components.manifest.cli_wrapper import ManifestCLI, ManifestList
from python_on_whales.utils import run
import requests
from typing import List
from stack_orchestrator.opts import opts
from stack_orchestrator.util import include_exclude_check, error_exit
from stack_orchestrator.build.build_util import get_containers_in_scope
# Experimental fetch-container command
@dataclass
class RegistryInfo:
registry: str
registry_username: str
registry_token: str
# Extending this code to support the --verbose option, cnosider contributing upstream
# https://github.com/gabrieldemarmiesse/python-on-whales/blob/master/python_on_whales/components/manifest/cli_wrapper.py#L129
class ExtendedManifestCLI(ManifestCLI):
def inspect_verbose(self, x: str) -> ManifestList:
"""Returns a Docker manifest list object."""
json_str = run(self.docker_cmd + ["manifest", "inspect", "--verbose", x])
return json.loads(json_str)
def _local_tag_for(container: str):
return f"{container}:local"
# See: https://docker-docs.uclv.cu/registry/spec/api/
# Emulate this:
# $ curl -u "my-username:my-token" -X GET "https://<container-registry-hostname>/v2/cerc-io/cerc/test-container/tags/list"
# {"name":"cerc-io/cerc/test-container","tags":["202402232130","202402232208"]}
def _get_tags_for_container(container: str, registry_info: RegistryInfo) -> List[str]:
# registry looks like: git.vdb.to/cerc-io
registry_parts = registry_info.registry.split("/")
url = f"https://{registry_parts[0]}/v2/{registry_parts[1]}/{container}/tags/list"
if opts.o.debug:
print(f"Fetching tags from: {url}")
response = requests.get(url, auth=(registry_info.registry_username, registry_info.registry_token))
if response.status_code == 200:
tag_info = response.json()
if opts.o.debug:
print(f"container tags list: {tag_info}")
tags_array = tag_info["tags"]
return tags_array
else:
error_exit(f"failed to fetch tags from image registry, status code: {response.status_code}")
def _find_latest(candidate_tags: List[str]):
# Lex sort should give us the latest first
sorted_candidates = sorted(candidate_tags)
if opts.o.debug:
print(f"sorted candidates: {sorted_candidates}")
return sorted_candidates[-1]
def _filter_for_platform(container: str,
registry_info: RegistryInfo,
tag_list: List[str]) -> List[str] :
filtered_tags = []
this_machine = platform.machine()
# Translate between Python and docker platform names
if this_machine == "x86_64":
this_machine = "amd64"
if this_machine == "aarch64":
this_machine = "arm64"
if opts.o.debug:
print(f"Python says the architecture is: {this_machine}")
docker = DockerClient()
for tag in tag_list:
remote_tag = f"{registry_info.registry}/{container}:{tag}"
manifest_cmd = ExtendedManifestCLI(docker.client_config)
manifest = manifest_cmd.inspect_verbose(remote_tag)
if opts.o.debug:
print(f"manifest: {manifest}")
image_architecture = manifest["Descriptor"]["platform"]["architecture"]
if opts.o.debug:
print(f"image_architecture: {image_architecture}")
if this_machine == image_architecture:
filtered_tags.append(tag)
if opts.o.debug:
print(f"Tags filtered for platform: {filtered_tags}")
return filtered_tags
def _get_latest_image(container: str, registry_info: RegistryInfo):
all_tags = _get_tags_for_container(container, registry_info)
tags_for_platform = _filter_for_platform(container, registry_info, all_tags)
if len(tags_for_platform) > 0:
latest_tag = _find_latest(tags_for_platform)
return f"{container}:{latest_tag}"
else:
return None
def _fetch_image(tag: str, registry_info: RegistryInfo):
docker = DockerClient()
remote_tag = f"{registry_info.registry}/{tag}"
if opts.o.debug:
print(f"Attempting to pull this image: {remote_tag}")
docker.image.pull(remote_tag)
def _exists_locally(container: str):
docker = DockerClient()
return docker.image.exists(_local_tag_for(container))
def _add_local_tag(remote_tag: str, registry: str, local_tag: str):
docker = DockerClient()
docker.image.tag(f"{registry}/{remote_tag}", local_tag)
@click.command()
@click.option('--include', help="only fetch these containers")
@click.option('--exclude', help="don\'t fetch these containers")
@click.option("--force-local-overwrite", is_flag=True, default=False, help="Overwrite a locally built image, if present")
@click.option("--image-registry", required=True, help="Specify the image registry to fetch from")
@click.option("--registry-username", required=True, help="Specify the image registry username")
@click.option("--registry-token", required=True, help="Specify the image registry access token")
@click.pass_context
def command(ctx, include, exclude, force_local_overwrite, image_registry, registry_username, registry_token):
'''EXPERIMENTAL: fetch the images for a stack from remote registry'''
registry_info = RegistryInfo(image_registry, registry_username, registry_token)
docker = DockerClient()
if not opts.o.quiet:
print("Logging into container registry:")
docker.login(registry_info.registry, registry_info.registry_username, registry_info.registry_token)
# Generate list of target containers
stack = ctx.obj.stack
containers_in_scope = get_containers_in_scope(stack)
all_containers_found = True
for container in containers_in_scope:
local_tag = _local_tag_for(container)
if include_exclude_check(container, include, exclude):
if opts.o.debug:
print(f"Processing: {container}")
# For each container, attempt to find the latest of a set of
# images with the correct name and platform in the specified registry
image_to_fetch = _get_latest_image(container, registry_info)
if not image_to_fetch:
print(f"Warning: no image found to fetch for container: {container}")
all_containers_found = False
continue
if opts.o.debug:
print(f"Fetching: {image_to_fetch}")
_fetch_image(image_to_fetch, registry_info)
# Now check if the target container already exists exists locally already
if (_exists_locally(container)):
if not opts.o.quiet:
print(f"Container image {container} already exists locally")
# if so, fail unless the user specified force-local-overwrite
if (force_local_overwrite):
# In that case remove the existing :local tag
if not opts.o.quiet:
print(f"Warning: overwriting local tag from this image: {container} because "
"--force-local-overwrite was specified")
else:
if not opts.o.quiet:
print(f"Skipping local tagging for this image: {container} because that would "
"overwrite an existing :local tagged image, use --force-local-overwrite to do so.")
continue
# Tag the fetched image with the :local tag
_add_local_tag(image_to_fetch, image_registry, local_tag)
else:
if opts.o.verbose:
print(f"Excluding: {container}")
if not all_containers_found:
print("Warning: couldn't find usable images for one or more containers, this stack will not deploy")
+48
View File
@@ -0,0 +1,48 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
from datetime import datetime
from python_on_whales import DockerClient
from stack_orchestrator.opts import opts
from stack_orchestrator.util import error_exit
def _publish_tag_for_image(local_image_tag: str, remote_repo: str, version: str):
# Turns image tags of the form: foo/bar:local into remote.repo/org/bar:deploy
(image_name, image_version) = local_image_tag.split(":")
if image_version == "local":
return f"{remote_repo}/{image_name}:{version}"
else:
error_exit("Asked to publish a non-locally built image")
def publish_image(local_tag, registry):
if opts.o.verbose:
print(f"Publishing this image: {local_tag} to this registry: {registry}")
docker = DockerClient()
# Figure out the target image tag
# Eventually this version will be generated from the source repo state
# Using a timestemp is an intermediate step
version = datetime.now().strftime("%Y%m%d%H%M")
remote_tag = _publish_tag_for_image(local_tag, registry, version)
# Tag the image thus
if opts.o.debug:
print(f"Tagging {local_tag} to {remote_tag}")
docker.image.tag(local_tag, remote_tag)
# Push it to the desired registry
if opts.o.verbose:
print(f"Pushing image {remote_tag}")
docker.image.push(remote_tag)
+7 -1
View File
@@ -27,6 +27,12 @@ kube_config_key = "kube-config"
deploy_to_key = "deploy-to"
network_key = "network"
http_proxy_key = "http-proxy"
image_resigtry_key = "image-registry"
image_registry_key = "image-registry"
configmaps_key = "configmaps"
resources_key = "resources"
volumes_key = "volumes"
security_key = "security"
annotations_key = "annotations"
labels_key = "labels"
kind_config_filename = "kind-config.yml"
kube_config_filename = "kubeconfig.yml"
@@ -0,0 +1,15 @@
services:
registry:
image: registry:2.8
restart: always
environment:
REGISTRY_LOG_LEVEL: ${REGISTRY_LOG_LEVEL}
volumes:
- config:/config:ro
- registry-data:/var/lib/registry
ports:
- "5000"
volumes:
config:
registry-data:
@@ -26,9 +26,9 @@ services:
- "80"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "80"]
interval: 20s
interval: 10s
timeout: 5s
retries: 15
retries: 30
start_period: 10s
volumes:
@@ -0,0 +1,12 @@
version: "3.2"
services:
mars:
image: cerc/mars-v2:local
restart: always
ports:
- "3000:3000"
environment:
- URL_OSMOSIS_REST=https://lcd-osmosis.blockapsis.com
- URL_OSMOSIS_RPC=https://rpc-osmosis.blockapsis.com
- WALLET_CONNECT_ID=0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x
@@ -0,0 +1,20 @@
version: "3.2"
services:
mars:
image: cerc/mars:local
restart: always
ports:
- "3000:3000"
environment:
- URL_OSMOSIS_GQL=https://osmosis-node.marsprotocol.io/GGSFGSFGFG34/osmosis-hive-front/graphql
- URL_OSMOSIS_REST=https://lcd-osmosis.blockapsis.com
- URL_OSMOSIS_RPC=https://rpc-osmosis.blockapsis.com
- URL_NEUTRON_GQL=https://neutron.rpc.p2p.world/qgrnU6PsQZA8F9S5Fb8Fn3tV3kXmMBl2M9bcc9jWLjQy8p/hive/graphql
- URL_NEUTRON_REST=https://rest-kralum.neutron-1.neutron.org
- URL_NEUTRON_RPC=https://rpc-kralum.neutron-1.neutron.org
- URL_NEUTRON_TEST_GQL=https://testnet-neutron-gql.marsprotocol.io/graphql
- URL_NEUTRON_TEST_REST=https://rest-palvus.pion-1.ntrn.tech
- URL_NEUTRON_TEST_RPC=https://rpc-palvus.pion-1.ntrn.tech
- WALLET_CONNECT_ID=0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x
@@ -0,0 +1,8 @@
version: "3.2"
services:
ping-pub:
image: cerc/ping-pub:local
restart: always
ports:
- "5173:5173"
@@ -0,0 +1,13 @@
services:
snowballtools-base-backend:
image: cerc/snowballtools-base-backend:local
restart: always
volumes:
- data:/data
- config:/config:ro
ports:
- 8000
volumes:
data:
config:
@@ -0,0 +1,20 @@
services:
database:
image: cerc/test-database-container:local
restart: always
volumes:
- db-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: "test-user"
POSTGRES_DB: "test-db"
POSTGRES_PASSWORD: "password"
POSTGRES_INITDB_ARGS: "-E UTF8 --locale=C"
ports:
- "5432"
test-client:
image: cerc/test-database-client:local
volumes:
db-data:
@@ -5,10 +5,16 @@ services:
environment:
CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
CERC_TEST_PARAM_1: ${CERC_TEST_PARAM_1:-FAILED}
CERC_TEST_PARAM_2: "CERC_TEST_PARAM_2_VALUE"
CERC_TEST_PARAM_3: ${CERC_TEST_PARAM_3:-FAILED}
volumes:
- test-data:/data
- test-data-bind:/data
- test-data-auto:/data2
- test-config:/config:ro
ports:
- "80"
volumes:
test-data:
test-data-bind:
test-data-auto:
test-config:
@@ -7,12 +7,12 @@ services:
environment:
- REACT_APP_INFURA_KEY=${CERC_INFURA_KEY}
- REACT_APP_AWS_API_ENDPOINT=${CERC_UNISWAP_GQL}
- REACT_APP_UNISWAP_API_URL=${CERC_UNISWAP_API}
command: ["./build-app.sh"]
volumes:
- ../config/uniswap-interface/build-app.sh:/app/build-app.sh
- urbit_app_builds:/app-builds
- ../config/uniswap-interface/urbit-files/mar:/app/mar
- ../config/uniswap-interface/urbit-files/desk.docket-0:/app/desk.docket-0
- ../config/uniswap-interface/urbit-files:/app/urbit-files
volumes:
urbit_app_builds:
@@ -0,0 +1,19 @@
version: "3.2"
# This is a short-lived container. It runs the Vega build and then copies the output to the
# `urbit_app_builds` volume.
services:
vega-interface:
image: cerc/vega-interface:local
restart: on-failure
# environment:
# - REACT_APP_INFURA_KEY=${CERC_INFURA_KEY}
# - REACT_APP_AWS_API_ENDPOINT=${CERC_UNISWAP_GQL}
entrypoint: ["./build-app.sh"]
volumes:
- ../config/vega-interface/urbit-files:/app/urbit-files
- ../config/vega-interface/build-app.sh:/app/build-app.sh
- urbit_app_builds:/app-builds
volumes:
urbit_app_builds:
@@ -0,0 +1,76 @@
version: '3.2'
services:
ajna-watcher-db:
restart: unless-stopped
image: postgres:14-alpine
environment:
- POSTGRES_USER=vdbm
- POSTGRES_MULTIPLE_DATABASES=ajna-watcher,ajna-watcher-job-queue
- POSTGRES_EXTENSION=ajna-watcher-job-queue:pgcrypto
- POSTGRES_PASSWORD=password
volumes:
- ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
- ajna_watcher_db_data:/var/lib/postgresql/data
ports:
- "5432"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "5432"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
ajna-watcher-job-runner:
restart: unless-stopped
depends_on:
ajna-watcher-db:
condition: service_healthy
image: cerc/watcher-ajna:local
environment:
CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
command: ["bash", "./start-job-runner.sh"]
volumes:
- ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
- ../config/watcher-ajna/start-job-runner.sh:/app/start-job-runner.sh
ports:
- "9000"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "9000"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s
extra_hosts:
- "host.docker.internal:host-gateway"
ajna-watcher-server:
restart: unless-stopped
depends_on:
ajna-watcher-db:
condition: service_healthy
ajna-watcher-job-runner:
condition: service_healthy
image: cerc/watcher-ajna:local
environment:
CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
command: ["bash", "./start-server.sh"]
volumes:
- ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
- ../config/watcher-ajna/start-server.sh:/app/start-server.sh
ports:
- "3008"
- "9001"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3008"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
ajna_watcher_db_data:
@@ -5,4 +5,4 @@ services:
environment:
CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
ports:
- "3000"
- "80"
@@ -50,22 +50,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -142,22 +126,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="censures", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -234,22 +202,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="claims", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -326,22 +278,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="conditional_star_release", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -418,22 +354,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="delegated_sending", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -510,22 +430,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="ecliptic", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -602,22 +506,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -694,22 +582,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="azimuth", instance="polls", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -788,22 +660,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="sushi", instance="sushiswap", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -880,22 +736,6 @@ groups:
legendFormat: __auto
range: false
refId: latest_external
- refId: latest_indexed
relativeTimeRange:
from: 600
to: 0
datasourceUid: PBFA97CFB590B2093
model:
datasource:
type: prometheus
uid: PBFA97CFB590B2093
editorMode: code
expr: sync_status_block_number{job="sushi", instance="merkl_sushiswap", kind="latest_indexed"}
hide: false
instant: true
legendFormat: __auto
range: false
refId: latest_indexed
- refId: condition
relativeTimeRange:
from: 600
@@ -15,7 +15,5 @@ yarn build
# Copy over build and other files to app-builds for urbit deployment
mkdir -p /app-builds/uniswap
cp -r ./build /app-builds/uniswap/
cp -r mar /app-builds/uniswap/
cp desk.docket-0 /app-builds/uniswap/
cp -r urbit-files /app-builds/uniswap/
cp -r ./build /app-builds/uniswap/urbit-files
@@ -0,0 +1,225 @@
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::
:: Proxy server
:: ------------
::
:: This is a simple agent which accepts requests at the path given in the `on-init` arm-- in this
:: case, "/apps/uniswap-proxy/"-- and proxies them to the upstream, in this case,
:: "https://api.uniswap.org/".
::
:: For example, assuming your ship is running on `localhost:8080`, a request to the url
::
:: `http://localhost:8080/apps/uniswap-proxy/v1/graphql`
::
:: will be proxied to
::
:: `https://api.uniswap.org/v1/graphql`
::
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::
/+ dbug, server, default-agent
::
|%
+$ card card:agent:gall
+$ state
$% %0
==
--
::
:: Path config
|%
++ base-url-path /apps/uniswap-proxy :: HTTP path on the ship that this agent will bind to
++ upstream-api-url "https://api.uniswap.org/" :: Upstream HTTP api that requests should be proxied to
--
::
%- agent:dbug
::
=| state
=* state -
::
^- agent:gall
=<
|_ =bowl:gall
+* this .
default ~(. (default-agent this) bowl)
::
++ on-init
^- (quip card _this)
:_ this
:~
[%pass /eyre/connect %arvo %e %connect [~ base-url-path] %proxy]
==
::
++ on-poke
|= [=mark =vase]
^- (quip card _this)
:_ this :: We have no state
?+ mark (on-poke:default mark vase)
%handle-http-request
=/ [eyre-id=@ta inbound-req=inbound-request:eyre] !<([@ta =inbound-request:eyre] vase)
~& "Eyre id: {<eyre-id>}"
=/ old-req request:inbound-req
=/ new-req=request:http
%= old-req
:: Remove path prefix from the url, replace with Uniswap domain and path prefix
url (rewrite-path url:old-req)
:: Change the "Origin" header to match the upstream (this is hacky as fuck and will break eventually)
header-list (rewrite-headers header-list:old-req)
==
~& "Proxying {<url.old-req>} to {<url.new-req>}"
~& "Outgoing headers: {<header-list:new-req>}"
:~
[%pass /response/[eyre-id] %arvo %i %request new-req *outbound-config:iris]
==
==
::
++ on-arvo
|= [=wire =sign-arvo]
^- (quip card _this)
:_ this :: We don't have any state
?+ sign-arvo (on-arvo:default wire sign-arvo)
::
:: Arvo will respond when we initially connect to Eyre in `on-init`. We will accept (and ignore)
:: that and reject any other communications.
[%eyre %bound *]
~& "Got eyre bound: {<sign-arvo>}"
~
[%iris %http-response %finished *]
?+ wire (on-arvo:default wire sign-arvo)
[%response @ ~]
=/ original-eyre-id=@ta (snag 1 `(list @ta)`wire)
=/ resp=client-response:iris +:+:sign-arvo
?> ?= %finished -.resp
=/ resp-header=response-header:http response-header:resp
=/ the-octs=octs data:(need full-file:resp)
::
=/ thedata=@t q.the-octs
~& resp-header
~& "Proxied HTTP {<status-code:resp-header>}: {<thedata>}"
%+ give-simple-payload:app:server
original-eyre-id
:- %_(resp-header headers [['Content-type'^'application/json'] ~])
`the-octs
==
==
::
:: Each time Eyre pokes a request to us, it will subscribe for the response. We will just accept
:: those connections (wire = /http-response/[eyre-id]) and reject any others.
:: See: https://docs.urbit.org/system/kernel/eyre/reference/tasks#connect
++ on-watch
|= =path
^- (quip card _this)
?+ path (on-watch:default path)
[%http-response *]
`this
==
::
++ on-save on-save:default
++ on-load on-load:default
++ on-leave on-leave:default
++ on-peek on-peek:default
++ on-agent on-agent:default
++ on-fail on-fail:default
--
::
:: Helpers core
|%
:: Rewrites a path like "/apps/uniswap-proxy/v1/graphql" to "https://api.uniswap.org/v1/graphql".
:: `url-path` is a path, not a full URL!
++ rewrite-path
|= [url-path=@t]
%+ rash
url-path
:: Prepend the `upstream-api-url` base url and convert back to a cord
%+ cook
|= [a=tape]
(crip (weld upstream-api-url a))
:: Strip the base path
;~ pfix
(ifix [(just '/') (just '/')] (jest (crip (join '/' `(list @t)`base-url-path))))
(star prn)
==
::
:: Change the 'Origin' http header to "https://api.uniswap.org".
:: I don't know why this makes the request work, but it does.
:: Also drop the "x-forwarded-for" header for Red Horizon; otherwise this breaks it.
++ rewrite-headers
|= [headers=header-list:http]
^- header-list:http
=/ ret *header-list:http
|-
?~ headers
ret
%= $
ret ?+ (crip (cass (trip key:(head headers))))
::
:: Default: use the header
(snoc ret (head headers))
::
:: Drop "x-forwarded-for" header
%'x-forwarded-for'
ret
::
:: Drop the "cookie" header, which contains the Urbit auth token (!)
%'cookie'
ret
::
:: Rewrite "origin" header
%'origin'
(snoc ret ['origin' 'https://api.uniswap.org'])
==
headers +.headers
==
::
:: Manually construct a card that can be passed to iris, simulating a Uniswap graphql request.
:: Can be useful for testing purposes.
++ iris-request-card
::
:: The resulting card produces an HTTP request equivalent to the following cURL command:
::
:: curl 'https://api.uniswap.org/v1/graphql' \
:: -H 'authority: interface.gateway.uniswap.org' \
:: -H 'accept: */*' \
:: -H 'accept-language: en-US,en;q=0.7' \
:: -H 'content-type: application/json' \
:: -H 'origin: https://app.uniswap.org' \
:: -H 'referer: https://app.uniswap.org/' \
:: -H 'sec-ch-ua: "Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"' \
:: -H 'sec-ch-ua-mobile: ?0' \
:: -H 'sec-ch-ua-platform: "Linux"' \
:: -H 'sec-fetch-dest: empty' \
:: -H 'sec-fetch-mode: cors' \
:: -H 'sec-fetch-site: same-site' \
:: -H 'sec-gpc: 1' \
:: -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36' \
:: --data-raw $'{"operationName":"TokenSpotPrice","variables":{"address":null,"chain":"ETHEREUM"},"query":"query TokenSpotPrice($chain: Chain\u0021, $address: String = null) {\\n token(chain: $chain, address: $address) {\\n id\\n address\\n chain\\n name\\n symbol\\n project {\\n id\\n markets(currencies: [USD]) {\\n id\\n price {\\n id\\n value\\n __typename\\n }\\n __typename\\n }\\n __typename\\n }\\n __typename\\n }\\n}"}' \
:: --compressed
::
=/ the-request=request:http
:* %'POST' 'https://api.uniswap.org/v1/graphql'
:~ ['Accept-Language' 'en-US,en']
['Cache-Control' 'no-cache']
['Connection' 'keep-alive']
['Origin' 'http://localhost:8080']
['Pragma' 'no-cache']
['Referer' 'http://localhost:8080/']
['Sec-Fetch-Dest' 'empty']
['Sec-Fetch-Mode' 'cors']
['Sec-Fetch-Site' 'same-site']
['Sec-GPC' '1']
['User-Agent' 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36']
['accept' '*/*']
['content-type' 'application/json']
['sec-ch-ua' '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"']
['sec-ch-ua-mobile' '?0']
['sec-ch-ua-platform' '"Linux"']
==
:- ~ %- as-octs:mimes:html
'''
{"operationName":"TokenSpotPrice","variables":{"address":null,"chain":"ETHEREUM"},"query":"query TokenSpotPrice($chain: Chain\u0021, $address: String = null) {\n token(chain: $chain, address: $address) {\n id\n address\n chain\n name\n symbol\n project {\n id\n markets(currencies: [USD]) {\n id\n price {\n id\n value\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}"}
'''
==
:~
[%pass /response/1 %arvo %i %request the-request *outbound-config:iris]
==
--
@@ -0,0 +1,2 @@
:~ %proxy
==
@@ -0,0 +1,155 @@
:: dbug: agent wrapper for generic debugging tools
::
:: usage: %-(agent:dbug your-agent)
::
|%
+$ poke
$% [%bowl ~]
[%state grab=cord]
[%incoming =about]
[%outgoing =about]
==
::
+$ about
$@ ~
$% [%ship =ship]
[%path =path]
[%wire =wire]
[%term =term]
==
::
++ agent
|= =agent:gall
^- agent:gall
!.
|_ =bowl:gall
+* this .
ag ~(. agent bowl)
::
++ on-poke
|= [=mark =vase]
^- (quip card:agent:gall agent:gall)
?. ?=(%dbug mark)
=^ cards agent (on-poke:ag mark vase)
[cards this]
=/ dbug
!<(poke vase)
=; =tang
((%*(. slog pri 1) tang) [~ this])
?- -.dbug
%bowl [(sell !>(bowl))]~
::
%state
=? grab.dbug =('' grab.dbug) '-'
=; product=^vase
[(sell product)]~
=/ state=^vase
:: if the underlying app has implemented a /dbug/state scry endpoint,
:: use that vase in place of +on-save's.
::
=/ result=(each ^vase tang)
(mule |.(q:(need (need (on-peek:ag /x/dbug/state)))))
?:(?=(%& -.result) p.result on-save:ag)
%+ slap
(slop state !>([bowl=bowl ..zuse]))
(ream grab.dbug)
::
%incoming
=; =tang
?^ tang tang
[%leaf "no matching subscriptions"]~
%+ murn
%+ sort ~(tap by sup.bowl)
|= [[* a=[=ship =path]] [* b=[=ship =path]]]
(aor [path ship]:a [path ship]:b)
|= [=duct [=ship =path]]
^- (unit tank)
=; relevant=?
?. relevant ~
`>[path=path from=ship duct=duct]<
?: ?=(~ about.dbug) &
?- -.about.dbug
%ship =(ship ship.about.dbug)
%path ?=(^ (find path.about.dbug path))
%wire %+ lien duct
|=(=wire ?=(^ (find wire.about.dbug wire)))
%term !!
==
::
%outgoing
=; =tang
?^ tang tang
[%leaf "no matching subscriptions"]~
%+ murn
%+ sort ~(tap by wex.bowl)
|= [[[a=wire *] *] [[b=wire *] *]]
(aor a b)
|= [[=wire =ship =term] [acked=? =path]]
^- (unit tank)
=; relevant=?
?. relevant ~
`>[wire=wire agnt=[ship term] path=path ackd=acked]<
?: ?=(~ about.dbug) &
?- -.about.dbug
%ship =(ship ship.about.dbug)
%path ?=(^ (find path.about.dbug path))
%wire ?=(^ (find wire.about.dbug wire))
%term =(term term.about.dbug)
==
==
::
++ on-peek
|= =path
^- (unit (unit cage))
?. ?=([@ %dbug *] path)
(on-peek:ag path)
?+ path [~ ~]
[%u %dbug ~] ``noun+!>(&)
[%x %dbug %state ~] ``noun+!>(on-save:ag)
[%x %dbug %subscriptions ~] ``noun+!>([wex sup]:bowl)
==
::
++ on-init
^- (quip card:agent:gall agent:gall)
=^ cards agent on-init:ag
[cards this]
::
++ on-save on-save:ag
::
++ on-load
|= old-state=vase
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-load:ag old-state)
[cards this]
::
++ on-watch
|= =path
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-watch:ag path)
[cards this]
::
++ on-leave
|= =path
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-leave:ag path)
[cards this]
::
++ on-agent
|= [=wire =sign:agent:gall]
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-agent:ag wire sign)
[cards this]
::
++ on-arvo
|= [=wire =sign-arvo]
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-arvo:ag wire sign-arvo)
[cards this]
::
++ on-fail
|= [=term =tang]
^- (quip card:agent:gall agent:gall)
=^ cards agent (on-fail:ag term tang)
[cards this]
--
--
@@ -0,0 +1,65 @@
|= [agent=agent:gall]
|_ =bowl:gall
++ on-init
`agent
::
++ on-save
!>(~)
::
++ on-load
|= old-state=vase
`agent
::
++ on-poke
|= =cage
~| "unexpected poke to {<dap.bowl>} with mark {<p.cage>}"
!!
::
++ on-watch
|= =path
~| "unexpected subscription to {<dap.bowl>} on path {<path>}"
!!
::
++ on-leave
|= path
`agent
::
++ on-peek
|= =path
~| "unexpected scry into {<dap.bowl>} on path {<path>}"
!!
::
++ on-agent
|= [=wire =sign:agent:gall]
^- (quip card:agent:gall _agent)
?- -.sign
%poke-ack
?~ p.sign
`agent
%- (slog leaf+"poke failed from {<dap.bowl>} on wire {<wire>}" u.p.sign)
`agent
::
%watch-ack
?~ p.sign
`agent
=/ =tank leaf+"subscribe failed from {<dap.bowl>} on wire {<wire>}"
%- (slog tank u.p.sign)
`agent
::
%kick `agent
%fact
~| "unexpected subscription update to {<dap.bowl>} on wire {<wire>}"
~| "with mark {<p.cage.sign>}"
!!
==
::
++ on-arvo
|= [=wire =sign-arvo]
~| "unexpected system response {<sign-arvo>} to {<dap.bowl>} on wire {<wire>}"
!!
::
++ on-fail
|= [=term =tang]
%- (slog leaf+"error in {<dap.bowl>}" >term< tang)
`agent
--
@@ -0,0 +1,159 @@
=, eyre
|%
+$ request-line
$: [ext=(unit @ta) site=(list @t)]
args=(list [key=@t value=@t])
==
:: +parse-request-line: take a cord and parse out a url
::
++ parse-request-line
|= url=@t
^- request-line
(fall (rush url ;~(plug apat:de-purl:html yque:de-purl:html)) [[~ ~] ~])
::
++ manx-to-octs
|= man=manx
^- octs
(as-octt:mimes:html (en-xml:html man))
::
++ json-to-octs
|= jon=json
^- octs
(as-octs:mimes:html (en:json:html jon))
::
++ app
|%
::
:: +require-authorization:
:: redirect to the login page when unauthenticated
:: otherwise call handler on inbound request
::
++ require-authorization
|= $: =inbound-request:eyre
handler=$-(inbound-request:eyre simple-payload:http)
==
^- simple-payload:http
::
?: authenticated.inbound-request
~! this
~! +:*handler
(handler inbound-request)
::
=- [[307 ['location' -]~] ~]
%^ cat 3
'/~/login?redirect='
url.request.inbound-request
::
:: +require-authorization-simple:
:: redirect to the login page when unauthenticated
:: otherwise pass through simple-paylod
::
++ require-authorization-simple
|= [=inbound-request:eyre =simple-payload:http]
^- simple-payload:http
::
?: authenticated.inbound-request
~! this
simple-payload
::
=- [[307 ['location' -]~] ~]
%^ cat 3
'/~/login?redirect='
url.request.inbound-request
::
++ give-simple-payload
|= [eyre-id=@ta =simple-payload:http]
^- (list card:agent:gall)
=/ header-cage
[%http-response-header !>(response-header.simple-payload)]
=/ data-cage
[%http-response-data !>(data.simple-payload)]
:~ [%give %fact ~[/http-response/[eyre-id]] header-cage]
[%give %fact ~[/http-response/[eyre-id]] data-cage]
[%give %kick ~[/http-response/[eyre-id]] ~]
==
--
++ gen
|%
::
++ max-1-da ['cache-control' 'max-age=86400']
++ max-1-wk ['cache-control' 'max-age=604800']
::
++ html-response
=| cache=?
|= =octs
^- simple-payload:http
:_ `octs
[200 [['content-type' 'text/html'] ?:(cache [max-1-wk ~] ~)]]
::
++ css-response
=| cache=?
|= =octs
^- simple-payload:http
:_ `octs
[200 [['content-type' 'text/css'] ?:(cache [max-1-wk ~] ~)]]
::
++ js-response
=| cache=?
|= =octs
^- simple-payload:http
:_ `octs
[200 [['content-type' 'text/javascript'] ?:(cache [max-1-wk ~] ~)]]
::
++ png-response
=| cache=?
|= =octs
^- simple-payload:http
:_ `octs
[200 [['content-type' 'image/png'] ?:(cache [max-1-wk ~] ~)]]
::
++ svg-response
=| cache=?
|= =octs
^- simple-payload:http
:_ `octs
[200 [['content-type' 'image/svg+xml'] ?:(cache [max-1-wk ~] ~)]]
::
++ ico-response
|= =octs
^- simple-payload:http
[[200 [['content-type' 'image/x-icon'] max-1-wk ~]] `octs]
::
++ woff2-response
=| cache=?
|= =octs
^- simple-payload:http
[[200 [['content-type' 'font/woff2'] max-1-wk ~]] `octs]
::
++ json-response
=| cache=_|
|= =json
^- simple-payload:http
:_ `(json-to-octs json)
[200 [['content-type' 'application/json'] ?:(cache [max-1-da ~] ~)]]
::
++ manx-response
=| cache=_|
|= man=manx
^- simple-payload:http
:_ `(manx-to-octs man)
[200 [['content-type' 'text/html'] ?:(cache [max-1-da ~] ~)]]
::
++ not-found
^- simple-payload:http
[[404 ~] ~]
::
++ login-redirect
|= =request:http
^- simple-payload:http
=- [[307 ['location' -]~] ~]
%^ cat 3
'/~/login?redirect='
url.request
::
++ redirect
|= redirect=cord
^- simple-payload:http
[[307 ['location' redirect]~] ~]
--
--
@@ -0,0 +1,34 @@
|_ bil=(list dude:gall)
++ grow
|%
++ mime `^mime`[/text/x-bill (as-octs:mimes:html hoon)]
++ noun bil
++ hoon
^- @t
|^ (crip (of-wall:format (wrap-lines (spit-duz bil))))
::
++ wrap-lines
|= taz=wall
^- wall
?~ taz ["~"]~
:- (weld ":~ " i.taz)
%- snoc :_ "=="
(turn t.taz |=(t=tape (weld " " t)))
::
++ spit-duz
|= duz=(list dude:gall)
^- wall
(turn duz |=(=dude:gall ['%' (trip dude)]))
--
++ txt (to-wain:format hoon)
--
++ grab
|%
++ noun (list dude:gall)
++ mime
|= [=mite len=@ud tex=@]
~_ tex
!<((list dude:gall) (slap !>(~) (ream tex)))
--
++ grad %noun
--
@@ -0,0 +1,36 @@
:::: /hoon/hoon/mar
::
/? 310
::
=, eyre
|_ own=@t
::
++ grow :: convert to
|%
++ mime `^mime`[/text/x-hoon (as-octs:mimes:html own)] :: convert to %mime
++ hymn
;html
;head
;title:"Source"
;script@"//cdnjs.cloudflare.com/ajax/libs/codemirror/4.3.0/codemirror.js";
;script@"/lib/syntax/hoon.js";
;link(rel "stylesheet", href "//cdnjs.cloudflare.com/ajax/libs/".
"codemirror/4.3.0/codemirror.min.css");
;link/"/lib/syntax/codemirror.css"(rel "stylesheet");
==
;body
;textarea#src:"{(trip own)}"
;script:'CodeMirror.fromTextArea(src, {lineNumbers:true, readOnly:true})'
==
==
++ txt
(to-wain:format own)
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] q.q)
++ noun @t :: clam from %noun
++ txt of-wain:format
--
++ grad %txt
--
@@ -0,0 +1,28 @@
|_ kal=waft:clay
++ grow
|%
++ mime `^mime`[/text/x-kelvin (as-octs:mimes:html hoon)]
++ noun kal
++ hoon
%+ rap 3
%+ turn
%+ sort
~(tap in (waft-to-wefts:clay kal))
|= [a=weft b=weft]
?: =(lal.a lal.b)
(gte num.a num.b)
(gte lal.a lal.b)
|= =weft
(rap 3 '[%' (scot %tas lal.weft) ' ' (scot %ud num.weft) ']\0a' ~)
::
++ txt (to-wain:format hoon)
--
++ grab
|%
++ noun waft:clay
++ mime
|= [=mite len=@ud tex=@]
(cord-to-waft:clay tex)
--
++ grad %noun
--
@@ -1,18 +0,0 @@
::
:::: /hoon/map/mar
:: Mark for js source maps
/? 310
::
=, eyre
|_ mud=@
++ grow
|%
++ mime [/application/octet-stream (as-octs:mimes:html (@t mud))]
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] (@t q.q))
++ noun cord :: clam from %noun
--
++ grad %mime
--
@@ -0,0 +1,32 @@
::
:::: /hoon/mime/mar
::
/? 310
::
|_ own=mime
++ grow
^?
|%
++ jam `@`q.q.own
--
::
++ grab :: convert from
^?
|%
++ noun mime :: clam from %noun
++ tape
|=(a=_"" [/application/x-urb-unknown (as-octt:mimes:html a)])
--
++ grad
^?
|%
++ form %mime
++ diff |=(mime +<)
++ pact |=(mime +<)
++ join |=([mime mime] `(unit mime)`~)
++ mash
|= [[ship desk mime] [ship desk mime]]
^- mime
~|(%mime-mash !!)
--
--
@@ -0,0 +1,22 @@
::
:::: /hoon/noun/mar
::
/? 310
!:
:::: A minimal noun mark
|_ non=*
++ grab |%
++ noun *
--
++ grow |%
++ mime [/application/x-urb-jam (as-octs:mimes:html (jam non))]
--
++ grad
|%
++ form %noun
++ diff |=(* +<)
++ pact |=(* +<)
++ join |=([* *] *(unit *))
++ mash |=([[ship desk *] [ship desk *]] `*`~|(%noun-mash !!))
--
--
@@ -1,12 +0,0 @@
|_ dat=octs
++ grow
|%
++ mime [/font/ttf dat]
--
++ grab
|%
++ mime |=([=mite =octs] octs)
++ noun octs
--
++ grad %mime
--
@@ -0,0 +1,274 @@
::
:::: /hoon/txt/mar
::
/? 310
::
=, clay
=, differ
=, format
=, mimes:html
|_ txt=wain
::
++ grab :: convert from
|%
++ mime |=((pair mite octs) (to-wain q.q))
++ noun wain :: clam from %noun
--
++ grow
=> v=.
|%
++ mime => v [/text/plain (as-octs (of-wain txt))]
--
++ grad
|%
++ form %txt-diff
++ diff
|= tyt=wain
^- (urge cord)
(lusk txt tyt (loss txt tyt))
::
++ pact
|= dif=(urge cord)
~| [%pacting dif]
^- wain
(lurk txt dif)
::
++ join
|= [ali=(urge cord) bob=(urge cord)]
^- (unit (urge cord))
|^
=. ali (clean ali)
=. bob (clean bob)
|- ^- (unit (urge cord))
?~ ali `bob
?~ bob `ali
?- -.i.ali
%&
?- -.i.bob
%&
?: =(p.i.ali p.i.bob)
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
?: (gth p.i.ali p.i.bob)
%+ bind $(p.i.ali (sub p.i.ali p.i.bob), bob t.bob)
|=(cud=(urge cord) [i.bob cud])
%+ bind $(ali t.ali, p.i.bob (sub p.i.bob p.i.ali))
|=(cud=(urge cord) [i.ali cud])
::
%|
?: =(p.i.ali (lent p.i.bob))
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.bob cud])
?: (gth p.i.ali (lent p.i.bob))
%+ bind $(p.i.ali (sub p.i.ali (lent p.i.bob)), bob t.bob)
|=(cud=(urge cord) [i.bob cud])
~
==
::
%|
?- -.i.bob
%|
?. =(i.ali i.bob)
~
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
::
%&
?: =(p.i.bob (lent p.i.ali))
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
?: (gth p.i.bob (lent p.i.ali))
%+ bind $(ali t.ali, p.i.bob (sub p.i.bob (lent p.i.ali)))
|=(cud=(urge cord) [i.ali cud])
~
==
==
++ clean :: clean
|= wig=(urge cord)
^- (urge cord)
?~ wig ~
?~ t.wig wig
?: ?=(%& -.i.wig)
?: ?=(%& -.i.t.wig)
$(wig [[%& (add p.i.wig p.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
?: ?=(%| -.i.t.wig)
$(wig [[%| (welp p.i.wig p.i.t.wig) (welp q.i.wig q.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
--
::
++ mash
|= $: [als=ship ald=desk ali=(urge cord)]
[bos=ship bod=desk bob=(urge cord)]
==
^- (urge cord)
|^
=. ali (clean ali)
=. bob (clean bob)
|- ^- (urge cord)
?~ ali bob
?~ bob ali
?- -.i.ali
%&
?- -.i.bob
%&
?: =(p.i.ali p.i.bob)
[i.ali $(ali t.ali, bob t.bob)]
?: (gth p.i.ali p.i.bob)
[i.bob $(p.i.ali (sub p.i.ali p.i.bob), bob t.bob)]
[i.ali $(ali t.ali, p.i.bob (sub p.i.bob p.i.ali))]
::
%|
?: =(p.i.ali (lent p.i.bob))
[i.bob $(ali t.ali, bob t.bob)]
?: (gth p.i.ali (lent p.i.bob))
[i.bob $(p.i.ali (sub p.i.ali (lent p.i.bob)), bob t.bob)]
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
:: ~ :: here, alice is good for a while, but not for the whole
== :: length of bob's changes
::
%|
?- -.i.bob
%|
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
::
%&
?: =(p.i.bob (lent p.i.ali))
[i.ali $(ali t.ali, bob t.bob)]
?: (gth p.i.bob (lent p.i.ali))
[i.ali $(ali t.ali, p.i.bob (sub p.i.bob (lent p.i.ali)))]
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
==
==
::
++ annotate :: annotate conflict
|= $: ali=(list @t)
bob=(list @t)
bas=(list @t)
==
^- (list @t)
%- zing
^- (list (list @t))
%- flop
^- (list (list @t))
:- :_ ~
%^ cat 3 '<<<<<<<<<<<<'
%^ cat 3 ' '
%^ cat 3 `@t`(scot %p bos)
%^ cat 3 '/'
bod
:- bob
:- ~['------------']
:- bas
:- ~['++++++++++++']
:- ali
:- :_ ~
%^ cat 3 '>>>>>>>>>>>>'
%^ cat 3 ' '
%^ cat 3 `@t`(scot %p als)
%^ cat 3 '/'
ald
~
::
++ clean :: clean
|= wig=(urge cord)
^- (urge cord)
?~ wig ~
?~ t.wig wig
?: ?=(%& -.i.wig)
?: ?=(%& -.i.t.wig)
$(wig [[%& (add p.i.wig p.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
?: ?=(%| -.i.t.wig)
$(wig [[%| (welp p.i.wig p.i.t.wig) (welp q.i.wig q.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
::
++ resolve
|= [ali=(urge cord) bob=(urge cord)]
^- [fic=[%| p=(list cord) q=(list cord)] ali=(urge cord) bob=(urge cord)]
=- [[%| bac (annotate alc boc bac)] ali bob]
|- ^- $: $: bac=(list cord)
alc=(list cord)
boc=(list cord)
==
ali=(urge cord)
bob=(urge cord)
==
?~ ali [[~ ~ ~] ali bob]
?~ bob [[~ ~ ~] ali bob]
?- -.i.ali
%&
?- -.i.bob
%& [[~ ~ ~] ali bob] :: no conflict
%|
=+ lob=(lent p.i.bob)
?: =(lob p.i.ali)
[[p.i.bob p.i.bob q.i.bob] t.ali t.bob]
?: (lth lob p.i.ali)
[[p.i.bob p.i.bob q.i.bob] [[%& (sub p.i.ali lob)] t.ali] t.bob]
=+ wat=(scag (sub lob p.i.ali) p.i.bob)
=+ ^= res
%= $
ali t.ali
bob [[%| (scag (sub lob p.i.ali) p.i.bob) ~] t.bob]
==
:* :* (welp bac.res wat)
(welp alc.res wat)
(welp boc.res q.i.bob)
==
ali.res
bob.res
==
==
::
%|
?- -.i.bob
%&
=+ loa=(lent p.i.ali)
?: =(loa p.i.bob)
[[p.i.ali q.i.ali p.i.ali] t.ali t.bob]
?: (lth loa p.i.bob)
[[p.i.ali q.i.ali p.i.ali] t.ali [[%& (sub p.i.bob loa)] t.bob]]
=+ wat=(slag (sub loa p.i.bob) p.i.ali)
=+ ^= res
%= $
ali [[%| (scag (sub loa p.i.bob) p.i.ali) ~] t.ali]
bob t.bob
==
:* :* (welp bac.res wat)
(welp alc.res q.i.ali)
(welp boc.res wat)
==
ali.res
bob.res
==
::
%|
=+ loa=(lent p.i.ali)
=+ lob=(lent p.i.bob)
?: =(loa lob)
[[p.i.ali q.i.ali q.i.bob] t.ali t.bob]
=+ ^= res
?: (gth loa lob)
$(ali [[%| (scag (sub loa lob) p.i.ali) ~] t.ali], bob t.bob)
~& [%scagging loa=loa pibob=p.i.bob slag=(scag loa p.i.bob)]
$(ali t.ali, bob [[%| (scag (sub lob loa) p.i.bob) ~] t.bob])
:* :* (welp bac.res ?:((gth loa lob) p.i.bob p.i.ali))
(welp alc.res q.i.ali)
(welp boc.res q.i.bob)
==
ali.res
bob.res
==
==
==
--
--
--
@@ -1,12 +0,0 @@
|_ dat=octs
++ grow
|%
++ mime [/font/woff dat]
--
++ grab
|%
++ mime |=([=mite =octs] octs)
++ noun octs
--
++ grad %mime
--
@@ -17,16 +17,10 @@ if [ -d ${app_desk_dir} ]; then
exit 0
fi
app_build=/app-builds/${CERC_URBIT_APP}/build
app_mark_files=/app-builds/${CERC_URBIT_APP}/mar
app_docket_file=/app-builds/${CERC_URBIT_APP}/desk.docket-0
echo "Reading app build from ${app_build}"
echo "Reading additional mark files from ${app_mark_files}"
echo "Reading docket file ${app_docket_file}"
app_files=/app-builds/${CERC_URBIT_APP}/urbit-files
# Loop until the app's build appears
while [ ! -d ${app_build} ]; do
while [ ! -d "${app_files}/build" ]; do
echo "${CERC_URBIT_APP} app build not found, retrying in 5s..."
sleep 5
done
@@ -50,15 +44,16 @@ hood () {
hood "merge %${CERC_URBIT_APP} our %landscape"
hood "mount %${CERC_URBIT_APP}"
# Copy over build to desk data dir
cp -r ${app_build} ${app_desk_dir}
# Copy over the additional mark files
cp ${app_mark_files}/* ${app_desk_dir}/mar/
rm "${app_desk_dir}/desk.bill"
rm "${app_desk_dir}/desk.bill" # Don't install the landscape apps
rm "${app_desk_dir}/desk.ship"
echo "Copying files from ${app_files}"
# Copy over build to desk data dir
cp -r ${app_files}/* ${app_desk_dir}
rm ${app_desk_dir}/desk.docket-0 # Remove until we have a valid glob path; it's added back again below
# Commit changes and create a glob
hood "commit %${CERC_URBIT_APP}"
dojo "-landscape!make-glob %${CERC_URBIT_APP} /build"
@@ -99,8 +94,8 @@ while true; do
done
# Replace the docket file for app
# Substitue the glob URL and hash
cp ${app_docket_file} ${app_desk_dir}/
# Substitute the glob URL and hash
cp ${app_files}/desk.docket-0 ${app_desk_dir}/
sed -i "s|REPLACE_WITH_GLOB_URL|${glob_url}|g; s|REPLACE_WITH_GLOB_HASH|${glob_hash}|g" ${app_desk_dir}/desk.docket-0
# Commit changes and install the app
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
set -e
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
# Check and exit if a deployment already exists (on restarts)
if [ -d /app-builds/vega/build ]; then
echo "Build already exists, remove volume to rebuild"
exit 0
fi
./docker/prepare-dist.sh
# Post-process the output: rename files, e.g., `AlphaLyrae.woff2` => `alphalyrae.woff2`
while read filepath; do
filename=$(basename "$filepath")
dir=$(dirname "$filepath")
if echo "$filename" | grep -q "[A-Z]"; then
newfilename=$(tr '[:upper:]' '[:lower:]' <<< "$filename")
echo "Translating '$filepath' => $dir/$newfilename'"
mv "$filepath" "$dir/$newfilename"
find dist-result -type f | xargs sed -i "s/$filename/$newfilename/g"
fi
done < <(find dist-result/ -type f)
# Copy over dist-result and other files to `app-builds` volume for urbit deployment
mkdir -p /app-builds/vega
cp -r ./dist-result /app-builds/vega/build
cp -r urbit-files/* /app-builds/vega/
@@ -0,0 +1,10 @@
:~ title+'Vegas'
info+'vegas'
color+0xcd.75df
image+'https://vegaprotocol.eth.limo/favicon.ico'
base+'vegas'
glob-http+['REPLACE_WITH_GLOB_URL' REPLACE_WITH_GLOB_HASH]
version+[0 0 1]
website+'https://vega.xyz/'
license+'MIT'
==
@@ -0,0 +1,223 @@
/- *docket
|%
::
++ mime
|%
+$ draft
$: title=(unit @t)
info=(unit @t)
color=(unit @ux)
glob-http=(unit [=url hash=@uvH])
glob-ames=(unit [=ship hash=@uvH])
base=(unit term)
site=(unit path)
image=(unit url)
version=(unit version)
website=(unit url)
license=(unit cord)
==
::
++ finalize
|= =draft
^- (unit docket)
?~ title.draft ~
?~ info.draft ~
?~ color.draft ~
?~ version.draft ~
?~ website.draft ~
?~ license.draft ~
=/ href=(unit href)
?^ site.draft `[%site u.site.draft]
?~ base.draft ~
?^ glob-http.draft
`[%glob u.base hash.u.glob-http %http url.u.glob-http]:draft
?~ glob-ames.draft
~
`[%glob u.base hash.u.glob-ames %ames ship.u.glob-ames]:draft
?~ href ~
=, draft
:- ~
:* %1
u.title
u.info
u.color
u.href
image
u.version
u.website
u.license
==
::
++ from-clauses
=| =draft
|= cls=(list clause)
^- (unit docket)
=* loop $
?~ cls (finalize draft)
=* clause i.cls
=. draft
?- -.clause
%title draft(title `title.clause)
%info draft(info `info.clause)
%color draft(color `color.clause)
%glob-http draft(glob-http `[url hash]:clause)
%glob-ames draft(glob-ames `[ship hash]:clause)
%base draft(base `base.clause)
%site draft(site `path.clause)
%image draft(image `url.clause)
%version draft(version `version.clause)
%website draft(website `website.clause)
%license draft(license `license.clause)
==
loop(cls t.cls)
::
++ to-clauses
|= d=docket
^- (list clause)
%- zing
:~ :~ title+title.d
info+info.d
color+color.d
version+version.d
website+website.d
license+license.d
==
?~ image.d ~ ~[image+u.image.d]
?: ?=(%site -.href.d) ~[site+path.href.d]
=/ ref=glob-reference glob-reference.href.d
:~ base+base.href.d
?- -.location.ref
%http [%glob-http url.location.ref hash.ref]
%ames [%glob-ames ship.location.ref hash.ref]
== == ==
::
++ spit-clause
|= =clause
^- tape
%+ weld " {(trip -.clause)}+"
?+ -.clause "'{(trip +.clause)}'"
%color (scow %ux color.clause)
%site (spud path.clause)
::
%glob-http
"['{(trip url.clause)}' {(scow %uv hash.clause)}]"
::
%glob-ames
"[{(scow %p ship.clause)} {(scow %uv hash.clause)}]"
::
%version
=, version.clause
"[{(scow %ud major)} {(scow %ud minor)} {(scow %ud patch)}]"
==
::
++ spit-docket
|= dock=docket
^- tape
;: welp
":~\0a"
`tape`(zing (join "\0a" (turn (to-clauses dock) spit-clause)))
"\0a=="
==
--
::
++ enjs
=, enjs:format
|%
::
++ charge-update
|= u=^charge-update
^- json
%+ frond -.u
^- json
?- -.u
%del-charge s+desk.u
::
%initial
%- pairs
%+ turn ~(tap by initial.u)
|=([=desk c=^charge] [desk (charge c)])
::
%add-charge
%- pairs
:~ desk+s+desk.u
charge+(charge charge.u)
==
==
::
++ num
|= a=@u
^- ^tape
=/ p=json (numb a)
?> ?=(%n -.p)
(trip p.p)
::
++ version
|= v=^version
^- json
:- %s
%- crip
"{(num major.v)}.{(num minor.v)}.{(num patch.v)}"
::
++ merge
|= [a=json b=json]
^- json
?> &(?=(%o -.a) ?=(%o -.b))
[%o (~(uni by p.a) p.b)]
::
++ href
|= h=^href
%+ frond -.h
?- -.h
%site s+(spat path.h)
%glob
%- pairs
:~ base+s+base.h
glob-reference+(glob-reference glob-reference.h)
==
==
::
++ glob-reference
|= ref=^glob-reference
%- pairs
:~ hash+s+(scot %uv hash.ref)
location+(glob-location location.ref)
==
::
++ glob-location
|= loc=^glob-location
^- json
%+ frond -.loc
?- -.loc
%http s+url.loc
%ames s+(scot %p ship.loc)
==
::
++ charge
|= c=^charge
%+ merge (docket docket.c)
%- pairs
:~ chad+(chad chad.c)
==
::
++ docket
|= d=^docket
^- json
%- pairs
:~ title+s+title.d
info+s+info.d
color+s+(scot %ux color.d)
href+(href href.d)
image+?~(image.d ~ s+u.image.d)
version+(version version.d)
license+s+license.d
website+s+website.d
==
::
++ chad
|= c=^chad
%+ frond -.c
?+ -.c ~
%hung s+err.c
==
--
--
@@ -0,0 +1,34 @@
|_ bil=(list dude:gall)
++ grow
|%
++ mime `^mime`[/text/x-bill (as-octs:mimes:html hoon)]
++ noun bil
++ hoon
^- @t
|^ (crip (of-wall:format (wrap-lines (spit-duz bil))))
::
++ wrap-lines
|= taz=wall
^- wall
?~ taz ["~"]~
:- (weld ":~ " i.taz)
%- snoc :_ "=="
(turn t.taz |=(t=tape (weld " " t)))
::
++ spit-duz
|= duz=(list dude:gall)
^- wall
(turn duz |=(=dude:gall ['%' (trip dude)]))
--
++ txt (to-wain:format hoon)
--
++ grab
|%
++ noun (list dude:gall)
++ mime
|= [=mite len=@ud tex=@]
~_ tex
!<((list dude:gall) (slap !>(~) (ream tex)))
--
++ grad %noun
--
@@ -0,0 +1,25 @@
::
:::: /hoon/css/mar
::
/? 310
=, eyre
=, mimes:html
|_ mud=@t
++ grow :: convert to
|%
++ mime [/text/css (as-octs mud)] :: convert to %mime
++ hymn :: convert to %hymn
|^ html
++ style ;style
;- (trip mud)
==
++ html ;html:(head:"{style}" body)
--
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] (@t q.q))
++ noun @t :: clam from %noun
--
++ grad %mime
--
@@ -0,0 +1,25 @@
/+ dock=docket
|_ =docket:dock
++ grow
|%
++ mime
^- ^mime
[/text/x-docket (as-octt:mimes:html (spit-docket:mime:dock docket))]
++ noun docket
++ json (docket:enjs:dock docket)
--
++ grab
|%
::
++ mime
|= [=mite len=@ud tex=@]
^- docket:dock
%- need
%- from-clauses:mime:dock
!<((list clause:dock) (slap !>(~) (ream tex)))
::
++ noun docket:dock
--
++ grad %noun
--
@@ -0,0 +1,36 @@
:::: /hoon/hoon/mar
::
/? 310
::
=, eyre
|_ own=@t
::
++ grow :: convert to
|%
++ mime `^mime`[/text/x-hoon (as-octs:mimes:html own)] :: convert to %mime
++ hymn
;html
;head
;title:"Source"
;script@"//cdnjs.cloudflare.com/ajax/libs/codemirror/4.3.0/codemirror.js";
;script@"/lib/syntax/hoon.js";
;link(rel "stylesheet", href "//cdnjs.cloudflare.com/ajax/libs/".
"codemirror/4.3.0/codemirror.min.css");
;link/"/lib/syntax/codemirror.css"(rel "stylesheet");
==
;body
;textarea#src:"{(trip own)}"
;script:'CodeMirror.fromTextArea(src, {lineNumbers:true, readOnly:true})'
==
==
++ txt
(to-wain:format own)
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] q.q)
++ noun @t :: clam from %noun
++ txt of-wain:format
--
++ grad %txt
--
@@ -0,0 +1,22 @@
::
:::: /hoon/html/mar
::
/? 310
::
:::: compute
::
=, html
|_ htm=@t
++ grow :: convert to
^?
|% ::
++ mime [/text/html (met 3 htm) htm] :: to %mime
++ hymn (need (de-xml htm)) :: to %hymn
-- ::
++ grab ^?
|% :: convert from
++ noun @t :: clam from %noun
++ mime |=([p=mite q=octs] q.q) :: retrieve form %mime
--
++ grad %mime
--
@@ -0,0 +1,12 @@
|_ dat=@
++ grow
|%
++ mime [/image/x-icon (as-octs:mimes:html dat)]
--
++ grab
|%
++ mime |=([p=mite q=octs] q.q)
++ noun @
--
++ grad %mime
--
@@ -0,0 +1,12 @@
|_ dat=@
++ grow
|%
++ mime [/image/jpeg (as-octs:mimes:html dat)]
--
++ grab
|%
++ mime |=([p=mite q=octs] q.q)
++ noun @
--
++ grad %mime
--
@@ -0,0 +1,25 @@
::
:::: /hoon/js/mar
::
/? 310
::
=, eyre
|_ mud=@
++ grow
|%
++ mime [/application/javascript (as-octs:mimes:html (@t mud))]
++ hymn :: convert to %hymn
|^ html
++ script ;script
;- (trip (@t mud))
==
++ html ;html:(head:"{script}" body)
--
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] (@t q.q))
++ noun cord :: clam from %noun
--
++ grad %mime
--
@@ -0,0 +1,26 @@
::
:::: /hoon/json/mar
::
/? 310
::
:::: compute
::
=, eyre
=, format
=, html
|_ jon=^json
::
++ grow :: convert to
|%
++ mime [/application/json (as-octs:mimes -:txt)] :: convert to %mime
++ txt [(en:json jon)]~
--
++ grab
|% :: convert from
++ mime |=([p=mite q=octs] (fall (de:json (@t q.q)) *^json))
++ noun ^json :: clam from %noun
++ numb numb:enjs
++ time time:enjs
--
++ grad %mime
--
@@ -0,0 +1,42 @@
::
/- *json-rpc
::
|_ res=response
::
++ grad %noun
++ grow
|%
++ noun res
--
++ grab :: convert from
|%
++ noun response :: from noun
++ httr :: from httr
|= hit=httr:eyre
^- response
~| hit
?: ?=(%2 (div p.hit 100))
%- json
?~ r.hit
a+~
(need (de:json:html q:u.r.hit))
fail+hit
++ json :: from json
=, dejs-soft:format
|= a=json
^- response
=; dere
=+ res=((ar dere) a)
?~ res (need (dere a))
[%batch u.res]
|= a=json
^- (unit response)
=/ res=(unit [@t json])
::TODO breaks when no id present
((ot id+so result+some ~) a)
?^ res `[%result u.res]
~| a
:+ ~ %error %- need
((ot id+so error+(ot code+no message+so ~) ~) a)
--
--
@@ -0,0 +1,28 @@
|_ kal=waft:clay
++ grow
|%
++ mime `^mime`[/text/x-kelvin (as-octs:mimes:html hoon)]
++ noun kal
++ hoon
%+ rap 3
%+ turn
%+ sort
~(tap in (waft-to-wefts:clay kal))
|= [a=weft b=weft]
?: =(lal.a lal.b)
(gte num.a num.b)
(gte lal.a lal.b)
|= =weft
(rap 3 '[%' (scot %tas lal.weft) ' ' (scot %ud num.weft) ']\0a' ~)
::
++ txt (to-wain:format hoon)
--
++ grab
|%
++ noun waft:clay
++ mime
|= [=mite len=@ud tex=@]
(cord-to-waft:clay tex)
--
++ grad %noun
--
@@ -0,0 +1,32 @@
::
:::: /hoon/mime/mar
::
/? 310
::
|_ own=mime
++ grow
^?
|%
++ jam `@`q.q.own
--
::
++ grab :: convert from
^?
|%
++ noun mime :: clam from %noun
++ tape
|=(a=_"" [/application/x-urb-unknown (as-octt:mimes:html a)])
--
++ grad
^?
|%
++ form %mime
++ diff |=(mime +<)
++ pact |=(mime +<)
++ join |=([mime mime] `(unit mime)`~)
++ mash
|= [[ship desk mime] [ship desk mime]]
^- mime
~|(%mime-mash !!)
--
--
@@ -0,0 +1,22 @@
::
:::: /hoon/noun/mar
::
/? 310
!:
:::: A minimal noun mark
|_ non=*
++ grab |%
++ noun *
--
++ grow |%
++ mime [/application/x-urb-jam (as-octs:mimes:html (jam non))]
--
++ grad
|%
++ form %noun
++ diff |=(* +<)
++ pact |=(* +<)
++ join |=([* *] *(unit *))
++ mash |=([[ship desk *] [ship desk *]] `*`~|(%noun-mash !!))
--
--
@@ -0,0 +1,12 @@
|_ dat=@
++ grow
|%
++ mime [/image/png (as-octs:mimes:html dat)]
--
++ grab
|%
++ mime |=([p=mite q=octs] q.q)
++ noun @
--
++ grad %mime
--
@@ -0,0 +1,20 @@
|_ s=ship
++ grad %noun
++ grow
|%
++ noun s
++ json s+(scot %p s)
++ mime
^- ^mime
[/text/x-ship (as-octt:mimes:html (scow %p s))]
--
++ grab
|%
++ noun ship
++ json (su:dejs:format ;~(pfix sig fed:ag))
++ mime
|= [=mite len=@ tex=@]
(slav %p (snag 0 (to-wain:format tex)))
--
--
@@ -0,0 +1,16 @@
::
:::: /hoon/txt-diff/mar
::
/? 310
|_ txt-diff=(urge:clay cord)
::
++ grad %noun
++ grow
|%
++ noun txt-diff
--
++ grab :: convert from
|%
++ noun (urge:clay cord) :: make from %noun
--
--
@@ -0,0 +1,274 @@
::
:::: /hoon/txt/mar
::
/? 310
::
=, clay
=, differ
=, format
=, mimes:html
|_ txt=wain
::
++ grab :: convert from
|%
++ mime |=((pair mite octs) (to-wain q.q))
++ noun wain :: clam from %noun
--
++ grow
=> v=.
|%
++ mime => v [/text/plain (as-octs (of-wain txt))]
--
++ grad
|%
++ form %txt-diff
++ diff
|= tyt=wain
^- (urge cord)
(lusk txt tyt (loss txt tyt))
::
++ pact
|= dif=(urge cord)
~| [%pacting dif]
^- wain
(lurk txt dif)
::
++ join
|= [ali=(urge cord) bob=(urge cord)]
^- (unit (urge cord))
|^
=. ali (clean ali)
=. bob (clean bob)
|- ^- (unit (urge cord))
?~ ali `bob
?~ bob `ali
?- -.i.ali
%&
?- -.i.bob
%&
?: =(p.i.ali p.i.bob)
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
?: (gth p.i.ali p.i.bob)
%+ bind $(p.i.ali (sub p.i.ali p.i.bob), bob t.bob)
|=(cud=(urge cord) [i.bob cud])
%+ bind $(ali t.ali, p.i.bob (sub p.i.bob p.i.ali))
|=(cud=(urge cord) [i.ali cud])
::
%|
?: =(p.i.ali (lent p.i.bob))
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.bob cud])
?: (gth p.i.ali (lent p.i.bob))
%+ bind $(p.i.ali (sub p.i.ali (lent p.i.bob)), bob t.bob)
|=(cud=(urge cord) [i.bob cud])
~
==
::
%|
?- -.i.bob
%|
?. =(i.ali i.bob)
~
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
::
%&
?: =(p.i.bob (lent p.i.ali))
%+ bind $(ali t.ali, bob t.bob)
|=(cud=(urge cord) [i.ali cud])
?: (gth p.i.bob (lent p.i.ali))
%+ bind $(ali t.ali, p.i.bob (sub p.i.bob (lent p.i.ali)))
|=(cud=(urge cord) [i.ali cud])
~
==
==
++ clean :: clean
|= wig=(urge cord)
^- (urge cord)
?~ wig ~
?~ t.wig wig
?: ?=(%& -.i.wig)
?: ?=(%& -.i.t.wig)
$(wig [[%& (add p.i.wig p.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
?: ?=(%| -.i.t.wig)
$(wig [[%| (welp p.i.wig p.i.t.wig) (welp q.i.wig q.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
--
::
++ mash
|= $: [als=ship ald=desk ali=(urge cord)]
[bos=ship bod=desk bob=(urge cord)]
==
^- (urge cord)
|^
=. ali (clean ali)
=. bob (clean bob)
|- ^- (urge cord)
?~ ali bob
?~ bob ali
?- -.i.ali
%&
?- -.i.bob
%&
?: =(p.i.ali p.i.bob)
[i.ali $(ali t.ali, bob t.bob)]
?: (gth p.i.ali p.i.bob)
[i.bob $(p.i.ali (sub p.i.ali p.i.bob), bob t.bob)]
[i.ali $(ali t.ali, p.i.bob (sub p.i.bob p.i.ali))]
::
%|
?: =(p.i.ali (lent p.i.bob))
[i.bob $(ali t.ali, bob t.bob)]
?: (gth p.i.ali (lent p.i.bob))
[i.bob $(p.i.ali (sub p.i.ali (lent p.i.bob)), bob t.bob)]
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
:: ~ :: here, alice is good for a while, but not for the whole
== :: length of bob's changes
::
%|
?- -.i.bob
%|
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
::
%&
?: =(p.i.bob (lent p.i.ali))
[i.ali $(ali t.ali, bob t.bob)]
?: (gth p.i.bob (lent p.i.ali))
[i.ali $(ali t.ali, p.i.bob (sub p.i.bob (lent p.i.ali)))]
=/ [fic=(unce cord) ali=(urge cord) bob=(urge cord)]
(resolve ali bob)
[fic $(ali ali, bob bob)]
==
==
::
++ annotate :: annotate conflict
|= $: ali=(list @t)
bob=(list @t)
bas=(list @t)
==
^- (list @t)
%- zing
^- (list (list @t))
%- flop
^- (list (list @t))
:- :_ ~
%^ cat 3 '<<<<<<<<<<<<'
%^ cat 3 ' '
%^ cat 3 `@t`(scot %p bos)
%^ cat 3 '/'
bod
:- bob
:- ~['------------']
:- bas
:- ~['++++++++++++']
:- ali
:- :_ ~
%^ cat 3 '>>>>>>>>>>>>'
%^ cat 3 ' '
%^ cat 3 `@t`(scot %p als)
%^ cat 3 '/'
ald
~
::
++ clean :: clean
|= wig=(urge cord)
^- (urge cord)
?~ wig ~
?~ t.wig wig
?: ?=(%& -.i.wig)
?: ?=(%& -.i.t.wig)
$(wig [[%& (add p.i.wig p.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
?: ?=(%| -.i.t.wig)
$(wig [[%| (welp p.i.wig p.i.t.wig) (welp q.i.wig q.i.t.wig)] t.t.wig])
[i.wig $(wig t.wig)]
::
++ resolve
|= [ali=(urge cord) bob=(urge cord)]
^- [fic=[%| p=(list cord) q=(list cord)] ali=(urge cord) bob=(urge cord)]
=- [[%| bac (annotate alc boc bac)] ali bob]
|- ^- $: $: bac=(list cord)
alc=(list cord)
boc=(list cord)
==
ali=(urge cord)
bob=(urge cord)
==
?~ ali [[~ ~ ~] ali bob]
?~ bob [[~ ~ ~] ali bob]
?- -.i.ali
%&
?- -.i.bob
%& [[~ ~ ~] ali bob] :: no conflict
%|
=+ lob=(lent p.i.bob)
?: =(lob p.i.ali)
[[p.i.bob p.i.bob q.i.bob] t.ali t.bob]
?: (lth lob p.i.ali)
[[p.i.bob p.i.bob q.i.bob] [[%& (sub p.i.ali lob)] t.ali] t.bob]
=+ wat=(scag (sub lob p.i.ali) p.i.bob)
=+ ^= res
%= $
ali t.ali
bob [[%| (scag (sub lob p.i.ali) p.i.bob) ~] t.bob]
==
:* :* (welp bac.res wat)
(welp alc.res wat)
(welp boc.res q.i.bob)
==
ali.res
bob.res
==
==
::
%|
?- -.i.bob
%&
=+ loa=(lent p.i.ali)
?: =(loa p.i.bob)
[[p.i.ali q.i.ali p.i.ali] t.ali t.bob]
?: (lth loa p.i.bob)
[[p.i.ali q.i.ali p.i.ali] t.ali [[%& (sub p.i.bob loa)] t.bob]]
=+ wat=(slag (sub loa p.i.bob) p.i.ali)
=+ ^= res
%= $
ali [[%| (scag (sub loa p.i.bob) p.i.ali) ~] t.ali]
bob t.bob
==
:* :* (welp bac.res wat)
(welp alc.res q.i.ali)
(welp boc.res wat)
==
ali.res
bob.res
==
::
%|
=+ loa=(lent p.i.ali)
=+ lob=(lent p.i.bob)
?: =(loa lob)
[[p.i.ali q.i.ali q.i.bob] t.ali t.bob]
=+ ^= res
?: (gth loa lob)
$(ali [[%| (scag (sub loa lob) p.i.ali) ~] t.ali], bob t.bob)
~& [%scagging loa=loa pibob=p.i.bob slag=(scag loa p.i.bob)]
$(ali t.ali, bob [[%| (scag (sub lob loa) p.i.bob) ~] t.bob])
:* :* (welp bac.res ?:((gth loa lob) p.i.bob p.i.ali))
(welp alc.res q.i.ali)
(welp boc.res q.i.bob)
==
ali.res
bob.res
==
==
==
--
--
--
@@ -0,0 +1,12 @@
|_ dat=octs
++ grow
|%
++ mime [/font/woff dat]
--
++ grab
|%
++ mime |=([=mite =octs] octs)
++ noun octs
--
++ grad %mime
--
@@ -0,0 +1,12 @@
|_ dat=octs
++ grow
|%
++ mime [/font/woff2 dat]
--
++ grab
|%
++ mime |=([=mite =octs] octs)
++ noun octs
--
++ grad %mime
--
@@ -0,0 +1,5 @@
::
:::: /hoon/xhtml/mar
::
/= xhtml /mar/html
xhtml
@@ -0,0 +1,82 @@
|%
::
+$ version
[major=@ud minor=@ud patch=@ud]
::
+$ glob (map path mime)
::
+$ url cord
:: $glob-location: How to retrieve a glob
::
+$ glob-reference
[hash=@uvH location=glob-location]
::
+$ glob-location
$% [%http =url]
[%ames =ship]
==
:: $href: Where a tile links to
::
+$ href
$% [%glob base=term =glob-reference]
[%site =path]
==
:: $chad: State of a docket
::
+$ chad
$~ [%install ~]
$% :: Done
[%glob =glob]
[%site ~]
:: Waiting
[%install ~]
[%suspend glob=(unit glob)]
:: Error
[%hung err=cord]
==
::
:: $charge: A realized $docket
::
+$ charge
$: =docket
=chad
==
::
:: $clause: A key and value, as part of a docket
::
:: Only used to parse $docket
::
+$ clause
$% [%title title=@t]
[%info info=@t]
[%color color=@ux]
[%glob-http url=cord hash=@uvH]
[%glob-ames =ship hash=@uvH]
[%image =url]
[%site =path]
[%base base=term]
[%version =version]
[%website website=url]
[%license license=cord]
==
::
:: $docket: A description of JS bundles for a desk
::
+$ docket
$: %1
title=@t
info=@t
color=@ux
=href
image=(unit url)
=version
website=url
license=cord
==
::
+$ charge-update
$% [%initial initial=(map desk charge)]
[%add-charge =desk =charge]
[%del-charge =desk]
==
--
@@ -0,0 +1 @@
[%zuse 412]
@@ -0,0 +1,20 @@
#!/bin/sh
set -e
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
set -u
echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
# Read in the config template TOML file and modify it
WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
# Write the modified content to a new file
echo "$WATCHER_CONFIG" > environments/local.toml
echo "Running job-runner..."
DEBUG=vulcanize:* exec node --enable-source-maps dist/job-runner.js
@@ -0,0 +1,20 @@
#!/bin/sh
set -e
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
set -u
echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
# Read in the config template TOML file and modify it
WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
# Write the modified content to a new file
echo "$WATCHER_CONFIG" > environments/local.toml
echo "Running server..."
DEBUG=vulcanize:* exec node --enable-source-maps dist/server.js
@@ -0,0 +1,98 @@
[server]
host = "0.0.0.0"
port = 3008
kind = "active"
gqlPath = "/"
# Checkpointing state.
checkpointing = true
# Checkpoint interval in number of blocks.
checkpointInterval = 2000
# Enable state creation
# CAUTION: Disable only if state creation is not desired or can be filled subsequently
enableState = false
subgraphPath = "./subgraph-build"
# Interval to restart wasm instance periodically
wasmRestartBlocksInterval = 20
# Interval in number of blocks at which to clear entities cache.
clearEntitiesCacheInterval = 1000
# Max block range for which to return events in eventsInRange GQL query.
# Use -1 for skipping check on block range.
maxEventsBlockRange = 1000
# Flag to specify whether RPC endpoint supports block hash as block tag parameter
rpcSupportsBlockHashParam = false
# GQL cache settings
[server.gqlCache]
enabled = true
# Max in-memory cache size (in bytes) (default 8 MB)
# maxCacheSize
# GQL cache-control max-age settings (in seconds)
maxAge = 15
timeTravelMaxAge = 86400 # 1 day
[metrics]
host = "0.0.0.0"
port = 9000
[metrics.gql]
port = 9001
[database]
type = "postgres"
host = "ajna-watcher-db"
port = 5432
database = "ajna-watcher"
username = "vdbm"
password = "password"
synchronize = true
logging = false
[upstream]
[upstream.ethServer]
rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
# Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
rpcClient = true
# Boolean flag to specify if rpcProviderEndpoint is an FEVM RPC endpoint
isFEVM = true
# Boolean flag to filter event logs by contracts
filterLogsByAddresses = true
# Boolean flag to filter event logs by topics
filterLogsByTopics = true
[upstream.cache]
name = "requests"
enabled = false
deleteOnStart = false
[jobQueue]
dbConnectionString = "postgres://vdbm:password@ajna-watcher-db/ajna-watcher-job-queue"
maxCompletionLagInSecs = 300
jobDelayInMilliSecs = 100
eventsInBatch = 50
subgraphEventsOrder = true
# Filecoin block time: https://docs.filecoin.io/basics/the-blockchain/blocks-and-tipsets#blocktime
blockDelayInMilliSecs = 30000
# Boolean to switch between modes of processing events when starting the server.
# Setting to true will fetch filtered events and required blocks in a range of blocks and then process them.
# Setting to false will fetch blocks consecutively with its events and then process them (Behaviour is followed in realtime processing near head).
useBlockRanges = true
# Block range in which logs are fetched during historical blocks processing
historicalLogsBlockRange = 2000
# Max block range of historical processing after which it waits for completion of events processing
# If set to -1 historical processing does not wait for events processing and completes till latest canonical block
historicalMaxFetchAhead = 10000
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
# Build the mars-v2 image
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
docker build -t cerc/mars-v2:local -f ${CERC_REPO_BASE_DIR}/mars-v2-frontend/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/mars-v2-frontend
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
# Build the mars image
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
docker build -t cerc/mars:local -f ${CERC_REPO_BASE_DIR}/mars-interface/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/mars-interface
@@ -26,6 +26,8 @@ RUN \
&& su ${USERNAME} -c "umask 0002 && npm install -g eslint" \
# Install semver
&& su ${USERNAME} -c "umask 0002 && npm install -g semver" \
# Install pnpm
&& su ${USERNAME} -c "umask 0002 && npm install -g pnpm" \
&& npm cache clean --force > /dev/null 2>&1
# [Optional] Uncomment this section to install additional OS packages.
@@ -35,6 +37,9 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# [Optional] Uncomment if you want to install more global node modules
# RUN su node -c "npm install -g <your-package-list-here>"
# We do this to get a yq binary from the published container, for the correct architecture we're building here
COPY --from=docker.io/mikefarah/yq:latest /usr/bin/yq /usr/local/bin/yq
# Expose port for http
EXPOSE 80
@@ -11,8 +11,14 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/nextjs-base:local}
docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
rc=$?
if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
if [ $rc -ne 0 ]; then
echo "BUILD FAILED" 1>&2
exit $rc
fi
if [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
cat <<EOF
#################################################################
@@ -10,10 +10,12 @@ TRG_DIR="${3:-.next-r}"
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=npm
else
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
fi
fi
@@ -33,8 +35,8 @@ if [ -f ".env" ]; then
rm -f $TMP_ENV
fi
for f in $(find "$TRG_DIR" -regex ".*.[tj]sx?$" -type f | grep -v 'node_modules'); do
for e in $(cat "${f}" | tr -s '[:blank:]' '\n' | tr -s '[{},();]' '\n' | egrep -o '^"CERC_RUNTIME_ENV_[^\"]+"'); do
for f in $(find . -type f \( -regex '.*.html?' -or -regex ".*.[tj]s\(x\|on\)?$" \) | grep -v 'node_modules' | grep -v '.git'); do
for e in $(cat "${f}" | tr -s '[:blank:]' '\n' | tr -s '["/\\{},();]' '\n' | tr -s "[']" '\n' | egrep -o -e '^CERC_RUNTIME_ENV_.+$' -e '^LACONIC_HOSTED_CONFIG_.+$'); do
orig_name=$(echo -n "${e}" | sed 's/"//g')
cur_name=$(echo -n "${orig_name}" | sed 's/CERC_RUNTIME_ENV_//g')
cur_val=$(echo -n "\$${cur_name}" | envsubst)
@@ -9,7 +9,9 @@ CERC_MIN_NEXTVER=13.4.2
CERC_NEXT_VERSION="${CERC_NEXT_VERSION:-keep}"
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ]; then
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
@@ -113,7 +115,7 @@ if [ "$CERC_NEXT_VERSION" != "keep" ] && [ "$CUR_NEXT_VERSION" != "$CERC_NEXT_VE
mv package.json.$$ package.json
fi
$CERC_BUILD_TOOL install || exit 1
time $CERC_BUILD_TOOL install || exit 1
CUR_NEXT_VERSION=`jq -r '.version' node_modules/next/package.json`
@@ -136,9 +138,9 @@ to use for the build with:
EOF
cat package.json | jq ".dependencies.next = \"^$CERC_MIN_NEXTVER\"" > package.json.$$
mv package.json.$$ package.json
$CERC_BUILD_TOOL install || exit 1
time $CERC_BUILD_TOOL install || exit 1
fi
$CERC_BUILD_TOOL run cerc_compile || exit 1
time $CERC_BUILD_TOOL run cerc_compile || exit 1
exit 0
@@ -16,7 +16,9 @@ trap ctrl_c INT
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ] && [ ! -f "package-lock.json" ]; then
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Build the ping pub image
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
docker build -t cerc/ping-pub:local ${build_command_args} -f $CERC_REPO_BASE_DIR/explorer/Dockerfile $CERC_REPO_BASE_DIR/explorer
@@ -0,0 +1,6 @@
FROM cerc/snowballtools-base-backend-base:local
WORKDIR /app/packages/backend
COPY run.sh .
ENTRYPOINT ["./run.sh"]
@@ -0,0 +1,26 @@
FROM ubuntu:22.04 as builder
RUN apt update && \
apt install -y --no-install-recommends --no-install-suggests \
ca-certificates curl gnupg
# Node
ARG NODE_MAJOR=20
RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
apt update && apt install -y nodejs
# npm setup
RUN npm config set @cerc-io:registry https://git.vdb.to/api/packages/cerc-io/npm/ && npm install -g yarn
COPY . /app/
WORKDIR /app/
RUN find . -name 'node_modules' | xargs -n1 rm -rf
RUN yarn && yarn build --ignore frontend
FROM cerc/webapp-base:local
COPY --from=builder /app /app
WORKDIR /app/packages/backend
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
# Build cerc/webapp-deployer-backend
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/snowballtools-base-backend-base:local ${build_command_args} -f ${SCRIPT_DIR}/Dockerfile-base ${CERC_REPO_BASE_DIR}/snowballtools-base
docker build -t cerc/snowballtools-base-backend:local ${build_command_args} ${SCRIPT_DIR}
@@ -0,0 +1,19 @@
#!/bin/bash
LACONIC_HOSTED_CONFIG_FILE=${LACONIC_HOSTED_CONFIG_FILE}
if [ -z "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
if [ -f "/config/laconic-hosted-config.yml" ]; then
LACONIC_HOSTED_CONFIG_FILE="/config/laconic-hosted-config.yml"
elif [ -f "/config/config.yml" ]; then
LACONIC_HOSTED_CONFIG_FILE="/config/config.yml"
fi
fi
if [ -f "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
/scripts/apply-webapp-config.sh $LACONIC_HOSTED_CONFIG_FILE "`pwd`/dist"
fi
/scripts/apply-runtime-env.sh "`pwd`/dist"
yarn start
@@ -1,21 +1,67 @@
#!/usr/bin/env bash
set -e
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
# Test if the container's filesystem is old (run previously) or new
EXISTSFILENAME=/data/exists
echo "Test container starting"
if [[ -f "$EXISTSFILENAME" ]];
then
TIMESTAMP=`cat $EXISTSFILENAME`
echo "Filesystem is old, created: $TIMESTAMP"
DATA_DEVICE=$(df | grep "/data$" | awk '{ print $1 }')
if [[ -n "$DATA_DEVICE" ]]; then
echo "/data: MOUNTED dev=${DATA_DEVICE}"
else
echo "Filesystem is fresh"
echo `date` > $EXISTSFILENAME
echo "/data: not mounted"
fi
DATA2_DEVICE=$(df | grep "/data2$" | awk '{ print $1 }')
if [[ -n "$DATA_DEVICE" ]]; then
echo "/data2: MOUNTED dev=${DATA2_DEVICE}"
else
echo "/data2: not mounted"
fi
# Test if the container's filesystem is old (run previously) or new
for d in /data /data2; do
if [[ -f "$d/exists" ]];
then
TIMESTAMP=`cat $d/exists`
echo "$d filesystem is old, created: $TIMESTAMP"
else
echo "$d filesystem is fresh"
echo `date` > $d/exists
fi
done
if [ -n "$CERC_TEST_PARAM_1" ]; then
echo "Test-param-1: ${CERC_TEST_PARAM_1}"
fi
if [ -n "$CERC_TEST_PARAM_2" ]; then
echo "Test-param-2: ${CERC_TEST_PARAM_2}"
fi
if [ -n "$CERC_TEST_PARAM_3" ]; then
echo "Test-param-3: ${CERC_TEST_PARAM_3}"
fi
if [ -n "$CERC_TEST_PARAM_4" ]; then
echo "Test-param-4: ${CERC_TEST_PARAM_4}"
fi
if [ -n "$CERC_TEST_PARAM_5" ]; then
echo "Test-param-5: ${CERC_TEST_PARAM_5}"
fi
if [ -d "/config" ]; then
echo "/config: EXISTS"
for f in /config/*; do
if [[ -f "$f" ]] || [[ -L "$f" ]]; then
echo "$f:"
cat "$f"
echo ""
echo ""
fi
done
else
echo "/config: does NOT EXIST"
fi
# Run nginx which will block here forever
/usr/sbin/nginx -g "daemon off;"
@@ -0,0 +1,12 @@
FROM ubuntu:latest
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && export DEBCONF_NOWARNINGS="yes" && \
apt-get install -y software-properties-common && \
apt-get install -y postgresql-client && \
apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
EXPOSE 80
COPY run.sh /app/run.sh
ENTRYPOINT ["/app/run.sh"]
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Build cerc/test-container
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/test-database-client:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} $SCRIPT_DIR

Some files were not shown because too many files have changed in this diff Show More