Compare commits

...
42 Commits
Author SHA1 Message Date
telackey 500815fb1e Host ts-nitro example site through Caddy
Webapp Test / Run webapp test suite (pull_request) Failing after 0s
Smoke Test / Run basic test suite (pull_request) Failing after 0s
Lint Checks / Run linter (pull_request) Successful in 43s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m30s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m6s
2024-04-22 18:02:52 -05:00
telackey 4111e88d53 missing file
Deploy Test / Run deploy test suite (pull_request) Failing after 0s
Webapp Test / Run webapp test suite (pull_request) Failing after 0s
Smoke Test / Run basic test suite (pull_request) Failing after 0s
Lint Checks / Run linter (pull_request) Successful in 42s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m36s
2024-04-19 18:29:46 -05:00
telackey 659e13e2f8 empty file
Lint Checks / Run linter (pull_request) Successful in 54s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m27s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 7m34s
Webapp Test / Run webapp test suite (pull_request) Failing after 4m41s
Smoke Test / Run basic test suite (pull_request) Successful in 5m27s
2024-04-19 17:42:56 -05:00
telackey 781dd263ab Refactor to nitro-auth
Lint Checks / Run linter (pull_request) Successful in 33s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m17s
Webapp Test / Run webapp test suite (pull_request) Failing after 4m41s
Smoke Test / Run basic test suite (pull_request) Successful in 4m24s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 11m4s
2024-04-19 17:41:39 -05:00
telackey 7cce68959f relay
Lint Checks / Run linter (pull_request) Successful in 39s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m3s
Webapp Test / Run webapp test suite (pull_request) Failing after 4m15s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m52s
Smoke Test / Run basic test suite (pull_request) Successful in 4m19s
2024-04-04 22:55:45 -05:00
telackey 0cd90ad3dc Target reverse_proxy by env.
Lint Checks / Run linter (pull_request) Successful in 44s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m34s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m28s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 10m2s
Smoke Test / Run basic test suite (pull_request) Successful in 5m13s
2024-04-04 14:32:12 -05:00
telackey 90d92c3e15 Fix Docker build
Lint Checks / Run linter (pull_request) Successful in 50s
Webapp Test / Run webapp test suite (pull_request) Failing after 4m40s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m4s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m49s
Smoke Test / Run basic test suite (pull_request) Successful in 4m59s
2024-04-02 22:20:02 -05:00
telackey 202c68d996 unused file
Lint Checks / Run linter (pull_request) Successful in 43s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m31s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m17s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m5s
Smoke Test / Run basic test suite (pull_request) Successful in 4m33s
2024-04-02 21:25:06 -05:00
telackey 236dea1b44 typo
Lint Checks / Run linter (pull_request) Successful in 43s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m15s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m37s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m35s
Smoke Test / Run basic test suite (pull_request) Successful in 5m0s
2024-04-02 18:43:05 -05:00
telackey 9aa02fc26a missing env
Lint Checks / Run linter (pull_request) Successful in 42s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m55s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m24s
Smoke Test / Run basic test suite (pull_request) Successful in 4m31s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m40s
2024-04-02 18:39:48 -05:00
telackey 80e039d70e Remove abandoned file.
Lint Checks / Run linter (pull_request) Successful in 40s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m40s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m43s
Smoke Test / Run basic test suite (pull_request) Successful in 5m1s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m4s
2024-04-02 18:03:59 -05:00
telackey d7f06ad51e Combine stacks.
Lint Checks / Run linter (pull_request) Successful in 33s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m54s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m27s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m37s
Smoke Test / Run basic test suite (pull_request) Successful in 5m22s
2024-04-02 18:02:54 -05:00
telackey 343d31c324 Add README
Lint Checks / Run linter (pull_request) Successful in 43s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m10s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m33s
Smoke Test / Run basic test suite (pull_request) Successful in 4m56s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 10m13s
2024-04-01 16:07:47 -05:00
telackey fd00143b76 Add README
Lint Checks / Run linter (pull_request) Successful in 55s
Webapp Test / Run webapp test suite (pull_request) Failing after 4m55s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m51s
Smoke Test / Run basic test suite (pull_request) Successful in 5m36s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 10m57s
2024-04-01 16:06:52 -05:00
telackey 36e89b5a78 Linea Sepolia Testnet-based Nitro Auth
Lint Checks / Run linter (pull_request) Successful in 45s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m53s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m45s
Smoke Test / Run basic test suite (pull_request) Successful in 5m41s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m27s
2024-04-01 15:53:01 -05:00
telackey 80b299c1e5 Swap caddy for nginx
Lint Checks / Run linter (pull_request) Successful in 41s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m45s
Smoke Test / Run basic test suite (pull_request) Successful in 4m40s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m28s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 8m57s
2024-03-22 17:10:23 -05:00
telackey f20dcb8457 revert
Lint Checks / Run linter (pull_request) Successful in 39s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m35s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m16s
Smoke Test / Run basic test suite (pull_request) Successful in 5m14s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 11m8s
2024-03-21 16:19:42 -05:00
telackey 7929a137b6 revert
Lint Checks / Run linter (pull_request) Successful in 57s
Deploy Test / Run deploy test suite (pull_request) Successful in 6m39s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m35s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 10m57s
Smoke Test / Run basic test suite (pull_request) Successful in 6m19s
2024-03-21 16:19:14 -05:00
telackey 247025de28 Shorten sleep
Lint Checks / Run linter (pull_request) Successful in 52s
Deploy Test / Run deploy test suite (pull_request) Successful in 5m24s
Webapp Test / Run webapp test suite (pull_request) Successful in 4m36s
Smoke Test / Run basic test suite (pull_request) Successful in 6m15s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 11m40s
2024-03-21 12:12:26 -05:00
telackey 76821f00a0 stack updates 2024-03-21 02:10:37 -05:00
telackey 78219c0e04 Ask nitor 2024-03-19 21:11:52 -05:00
telackey 9e87088328 WIP: nitro auth 2024-03-19 16:49:51 -05:00
telackey 523b5779be Auto-detect which certificate to use (including wildcards). (#779)
Lint Checks / Run linter (push) Successful in 43s
Publish / Build and publish (push) Successful in 1m9s
Deploy Test / Run deploy test suite (push) Successful in 4m49s
Webapp Test / Run webapp test suite (push) Successful in 4m23s
Smoke Test / Run basic test suite (push) Successful in 5m5s
Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 9m18s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 54m22s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 55m24s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 7m53s
Database Test / Run database hosting test on kind/k8s (push) Successful in 11m14s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 4m22s
Rather than always requesting a certificate, attempt to re-use an existing certificate if it already exists in the k8s cluster.  This includes matching to a wildcard certificate.

Reviewed-on: #779
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-07 17:38:36 +00:00
telackey 62f7ce649d Exit non-0 if docker build fails. (#778)
Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 10m22s
Lint Checks / Run linter (push) Successful in 27s
Publish / Build and publish (push) Successful in 52s
Webapp Test / Run webapp test suite (push) Successful in 2m49s
Deploy Test / Run deploy test suite (push) Successful in 3m37s
Smoke Test / Run basic test suite (push) Successful in 2m43s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 37m29s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 2m52s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 8m43s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 50m50s
Make sure to check the exit code of the docker build and bubble it back up to laconic-so.

Reviewed-on: #778
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-06 18:38:30 +00:00
telackey cc541ac20f Use -slim variant for Dockerfile (#773)
Lint Checks / Run linter (push) Successful in 44s
Publish / Build and publish (push) Successful in 1m12s
Smoke Test / Run basic test suite (push) Successful in 2m48s
Webapp Test / Run webapp test suite (push) Successful in 5m40s
Deploy Test / Run deploy test suite (push) Successful in 6m23s
Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 9m42s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 4s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 35m37s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 5m21s
Database Test / Run database hosting test on kind/k8s (push) Successful in 7m12s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 2m50s
This saves about 1GB of space in the image.

Reviewed-on: #773
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-28 04:37:57 +00:00
telackey 10e2311a8b Add timed logging for the webapp build (#771)
Lint Checks / Run linter (push) Successful in 32s
Publish / Build and publish (push) Successful in 59s
Smoke Test / Run basic test suite (push) Successful in 3m1s
Webapp Test / Run webapp test suite (push) Successful in 4m47s
Deploy Test / Run deploy test suite (push) Successful in 5m40s
Add lots of log and timer output to webapp builds.

Reviewed-on: #771
2024-02-28 00:38:11 +00:00
dboreham f32bbf9e48 Merge pull request 'Doc for fetch-containers command' (#772) from dboreham/fetch-containers-doc into main
Lint Checks / Run linter (push) Successful in 31s
Publish / Build and publish (push) Successful in 1m15s
Webapp Test / Run webapp test suite (push) Successful in 4m9s
Smoke Test / Run basic test suite (push) Successful in 4m3s
Deploy Test / Run deploy test suite (push) Successful in 5m48s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 4m26s
Reviewed-on: #772
2024-02-27 18:45:18 +00:00
dboreham 0302153162 Doc for fetch-containers command
Lint Checks / Run linter (pull_request) Successful in 29s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m23s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m1s
Smoke Test / Run basic test suite (pull_request) Successful in 3m39s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m49s
2024-02-27 11:44:28 -07:00
dboreham 01e4437b62 Merge pull request 'Sort order was backwards' (#770) from dboreham/fix-container-age-sort into main
Lint Checks / Run linter (push) Successful in 30s
Publish / Build and publish (push) Successful in 1m3s
Deploy Test / Run deploy test suite (push) Successful in 3m51s
Webapp Test / Run webapp test suite (push) Successful in 2m58s
Smoke Test / Run basic test suite (push) Successful in 4m53s
Database Test / Run database hosting test on kind/k8s (push) Successful in 7m20s
Reviewed-on: #770
2024-02-27 16:01:17 +00:00
dboreham 64cec163b3 Sort order was backwards
Lint Checks / Run linter (pull_request) Successful in 32s
Webapp Test / Run webapp test suite (pull_request) Successful in 3m1s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m34s
Smoke Test / Run basic test suite (pull_request) Successful in 3m39s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 9m0s
2024-02-27 09:00:36 -07:00
dboreham 170ad71397 fetch-containers-fixes (#769)
Lint Checks / Run linter (push) Successful in 31s
Publish / Build and publish (push) Successful in 1m24s
Smoke Test / Run basic test suite (push) Has been cancelled
Deploy Test / Run deploy test suite (push) Has been cancelled
Webapp Test / Run webapp test suite (push) Has been cancelled
Reviewed-on: #769
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-27 15:53:05 +00:00
dboreham da1ff609fe fetch-images command (#768)
Lint Checks / Run linter (push) Successful in 35s
Publish / Build and publish (push) Successful in 1m33s
Webapp Test / Run webapp test suite (push) Successful in 3m44s
Smoke Test / Run basic test suite (push) Successful in 4m9s
Deploy Test / Run deploy test suite (push) Successful in 7m3s
Implementation of a command to fetch pre-built images from a remote registry, complementing the --push-images option already present on build-containers.

The two subcommands used together allow a stack to be deployed without needing to built its images, provided they have been already built and pushed to the specified container image registry.

This implementation simply picks the newest image with the right name and platform (matches against the platform Python is running on, so watch out for scenarios where Python is an x86 binary on M1 macs).

Reviewed-on: #768
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-27 15:15:08 +00:00
telackey 21eb9f036f Add support for pnpm as a webapp build tool. (#767)
Lint Checks / Run linter (push) Successful in 26s
Publish / Build and publish (push) Successful in 1m13s
Webapp Test / Run webapp test suite (push) Successful in 2m48s
Deploy Test / Run deploy test suite (push) Successful in 3m46s
Smoke Test / Run basic test suite (push) Successful in 5m1s
Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 9m43s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 48m33s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 55m22s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 8m48s
This adds support for auto-detecting pnpm as a build tool for webapps.

Reviewed-on: #767
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-26 23:31:52 +00:00
telackey a0413659f7 Check for existing tag in remote repo before building. (#764)
Lint Checks / Run linter (push) Successful in 59s
Publish / Build and publish (push) Successful in 44s
Webapp Test / Run webapp test suite (push) Successful in 2m49s
Deploy Test / Run deploy test suite (push) Successful in 5m41s
Smoke Test / Run basic test suite (push) Successful in 4m48s
Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 6m26s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Failing after 27m20s
Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 55m51s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 8m39s
Database Test / Run database hosting test on kind/k8s (push) Successful in 6m47s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 3m53s
webapps are meant to be build-once/deploy-many, but we were rebuilding them for every request.  This changes that, so that we rebuild only for every unique ApplicationRecord.

When we push the image, we now tag it according to its ApplicationRecord.

We don't want to use that tag directly in the compose file for the deployment, however, as the deployment needs to be able to adjust to new builds w/o re-writing the file all the time.  Instead, we use a per-deployment unique tag (same as before), we just update what image it references as needed.

Reviewed-on: #764
2024-02-24 03:22:49 +00:00
zramsayandzramsay a16fc657bf clarify uniswap urbit readme (#766)
Publish / Build and publish (push) Successful in 1m7s
Smoke Test / Run basic test suite (push) Successful in 3m27s
Deploy Test / Run deploy test suite (push) Successful in 4m28s
Webapp Test / Run webapp test suite (push) Successful in 4m30s
Lint Checks / Run linter (push) Successful in 29s
Co-authored-by: zramsay <zach@bluecollarcoding.ca>
Reviewed-on: #766
2024-02-24 00:15:53 +00:00
telackey 704c42c404 Use a catchall for single page apps. (#763)
Lint Checks / Run linter (push) Successful in 27s
Publish / Build and publish (push) Successful in 56s
Webapp Test / Run webapp test suite (push) Successful in 2m50s
Deploy Test / Run deploy test suite (push) Successful in 4m1s
Smoke Test / Run basic test suite (push) Successful in 6m18s
This creates a new environment variable, CERC_SINGLE_PAGE_APP, which controls whether a catchall redirection back to / is applied.

If the value is not explicitly set, we try to detect if the page looks like a single-page app.

Reviewed-on: #763
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-02-23 20:32:24 +00:00
dboreham 202f187172 Fix copy/paste error
Lint Checks / Run linter (push) Successful in 24s
Publish / Build and publish (push) Successful in 1m15s
Webapp Test / Run webapp test suite (push) Successful in 3m9s
Deploy Test / Run deploy test suite (push) Successful in 4m12s
Smoke Test / Run basic test suite (push) Successful in 5m15s
2024-02-23 13:15:37 -07:00
dboreham aaed356d32 Simple container image publication (#762)
Lint Checks / Run linter (push) Successful in 53s
Publish / Build and publish (push) Successful in 1m22s
Deploy Test / Run deploy test suite (push) Successful in 4m23s
Smoke Test / Run basic test suite (push) Successful in 2m57s
Webapp Test / Run webapp test suite (push) Successful in 4m47s
Reviewed-on: #762
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
2024-02-23 19:57:47 +00:00
telackey 2af6ffce77 Tweaks for running the container registry in k8s (#760)
Lint Checks / Run linter (push) Successful in 23s
Publish / Build and publish (push) Successful in 1m30s
Webapp Test / Run webapp test suite (push) Successful in 3m11s
Deploy Test / Run deploy test suite (push) Successful in 4m22s
Smoke Test / Run basic test suite (push) Successful in 4m57s
Minor tweaks for running the container-registry in k8s.  The big change is not requiring --image-registry.

Reviewed-on: #760
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
2024-02-22 21:11:06 +00:00
dboreham 7bb86cf35e Merge pull request 'Add ARM fixturenet-plugeth test job' (#759) from dboreham/arm-fixturenet-test into main
Lint Checks / Run linter (push) Successful in 44s
Publish / Build and publish (push) Successful in 1m16s
Smoke Test / Run basic test suite (push) Successful in 3m13s
Webapp Test / Run webapp test suite (push) Successful in 4m59s
Deploy Test / Run deploy test suite (push) Successful in 5m54s
Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 43m19s
Reviewed-on: #759
2024-02-22 20:45:26 +00:00
dboreham 9e0892cb6b No need to start docker now
Lint Checks / Run linter (pull_request) Successful in 57s
Deploy Test / Run deploy test suite (pull_request) Successful in 4m44s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Failing after 5m41s
Webapp Test / Run webapp test suite (pull_request) Successful in 5m48s
Smoke Test / Run basic test suite (pull_request) Successful in 5m38s
2024-02-22 13:36:00 -07:00
dboreham cf9cf6346f Add an arm-specific version of the plugeth fixturenet test 2024-02-22 13:34:37 -07:00
45 changed files with 1437 additions and 204 deletions
@@ -0,0 +1,45 @@
name: Fixturenet-Eth-Plugeth-Arm-Test
on:
push:
branches: '*'
paths:
- '!**'
- '.gitea/workflows/triggers/fixturenet-eth-plugeth-arm-test'
schedule: # Note: coordinate with other tests to not overload runners at the same time of day
- cron: '2 14 * * *'
# Needed until we can incorporate docker startup into the executor container
env:
DOCKER_HOST: unix:///var/run/dind.sock
jobs:
test:
name: "Run an Ethereum plugeth fixturenet test"
runs-on: ubuntu-latest-arm
steps:
- name: "Clone project repository"
uses: actions/checkout@v3
# At present the stock setup-python action fails on Linux/aarch64
# Conditional steps below workaroud this by using deadsnakes for that case only
- name: "Install Python for ARM on Linux"
if: ${{ runner.arch == 'arm64' && runner.os == 'Linux' }}
uses: deadsnakes/action@v3.0.1
with:
python-version: '3.8'
- name: "Install Python cases other than ARM on Linux"
if: ${{ ! (runner.arch == 'arm64' && runner.os == 'Linux') }}
uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: "Print Python version"
run: python3 --version
- name: "Install shiv"
run: pip install shiv
- name: "Generate build version file"
run: ./scripts/create_build_tag_file.sh
- name: "Build local shiv package"
run: ./scripts/build_shiv_package.sh
- name: "Run fixturenet-eth tests"
run: ./tests/fixturenet-eth-plugeth/run-test.sh
@@ -0,0 +1,2 @@
Change this file to trigger running the fixturenet-eth-plugeth-arm-test CI job
+9
View File
@@ -0,0 +1,9 @@
# Fetching pre-built container images
When Stack Orchestrator deploys a stack containing a suite of one or more containers it expects images for those containers to be on the local machine with a tag of the form `<image-name>:local` Images for these containers can be built from source (and optionally base container images from public registries) with the `build-containers` subcommand.
However, the task of building a large number of containers from source may consume considerable time and machine resources. This is where the `fetch-containers` subcommand steps in. It is designed to work exactly like `build-containers` but instead the images, pre-built, are fetched from an image registry then re-tagged for deployment. It can be used in place of `build-containers` for any stack provided the necessary containers, built for the local machine architecture (e.g. arm64 or x86-64) have already been published in an image registry.
## Usage
To use `fetch-containers`, provide an image registry path, a username and token/password with read access to the registry, and optionally specify `--force-local-overwrite`. If this argument is not specified, if there is already a locally built or previously fetched image for a stack container on the machine, it will not be overwritten and a warning issued.
```
$ laconic-so --stack mobymask-v3-demo fetch-containers --image-registry git.vdb.to/cerc-io --registry-username <registry-user> --registry-token <registry-token> --force-local-overwrite
```
+61 -68
View File
@@ -25,10 +25,13 @@ import sys
from decouple import config
import subprocess
import click
import importlib.resources
from pathlib import Path
from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config, stack_is_external, warn_exit
from stack_orchestrator.opts import opts
from stack_orchestrator.util import include_exclude_check, stack_is_external, error_exit
from stack_orchestrator.base import get_npm_registry_url
from stack_orchestrator.build.build_types import BuildContext
from stack_orchestrator.build.publish import publish_image
from stack_orchestrator.build.build_util import get_containers_in_scope
# TODO: find a place for this
# epilog="Config provided either in .env or settings.ini or env vars: CERC_REPO_BASE_DIR (defaults to ~/cerc)"
@@ -59,69 +62,58 @@ def make_container_build_env(dev_root_path: str,
return container_build_env
def process_container(stack: str,
container,
container_build_dir: str,
container_build_env: dict,
dev_root_path: str,
quiet: bool,
verbose: bool,
dry_run: bool,
continue_on_error: bool,
):
if not quiet:
print(f"Building: {container}")
def process_container(build_context: BuildContext) -> bool:
if not opts.o.quiet:
print(f"Building: {build_context.container}")
default_container_tag = f"{container}:local"
container_build_env.update({"CERC_DEFAULT_CONTAINER_IMAGE_TAG": default_container_tag})
default_container_tag = f"{build_context.container}:local"
build_context.container_build_env.update({"CERC_DEFAULT_CONTAINER_IMAGE_TAG": default_container_tag})
# Check if this is in an external stack
if stack_is_external(stack):
container_parent_dir = Path(stack).joinpath("container-build")
temp_build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
if stack_is_external(build_context.stack):
container_parent_dir = Path(build_context.stack).joinpath("container-build")
temp_build_dir = container_parent_dir.joinpath(build_context.container.replace("/", "-"))
temp_build_script_filename = temp_build_dir.joinpath("build.sh")
# Now check if the container exists in the external stack.
if not temp_build_script_filename.exists():
# If not, revert to building an internal container
container_parent_dir = container_build_dir
container_parent_dir = build_context.container_build_dir
else:
container_parent_dir = container_build_dir
container_parent_dir = build_context.container_build_dir
build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
build_dir = container_parent_dir.joinpath(build_context.container.replace("/", "-"))
build_script_filename = build_dir.joinpath("build.sh")
if verbose:
if opts.o.verbose:
print(f"Build script filename: {build_script_filename}")
if os.path.exists(build_script_filename):
build_command = build_script_filename.as_posix()
else:
if verbose:
if opts.o.verbose:
print(f"No script file found: {build_script_filename}, using default build script")
repo_dir = container.split('/')[1]
repo_dir = build_context.container.split('/')[1]
# TODO: make this less of a hack -- should be specified in some metadata somewhere
# Check if we have a repo for this container. If not, set the context dir to the container-build subdir
repo_full_path = os.path.join(dev_root_path, repo_dir)
repo_full_path = os.path.join(build_context.dev_root_path, repo_dir)
repo_dir_or_build_dir = repo_full_path if os.path.exists(repo_full_path) else build_dir
build_command = os.path.join(container_build_dir,
build_command = os.path.join(build_context.container_build_dir,
"default-build.sh") + f" {default_container_tag} {repo_dir_or_build_dir}"
if not dry_run:
if not opts.o.dry_run:
# No PATH at all causes failures with podman.
if "PATH" not in container_build_env:
container_build_env["PATH"] = os.environ["PATH"]
if verbose:
print(f"Executing: {build_command} with environment: {container_build_env}")
build_result = subprocess.run(build_command, shell=True, env=container_build_env)
if verbose:
if "PATH" not in build_context.container_build_env:
build_context.container_build_env["PATH"] = os.environ["PATH"]
if opts.o.verbose:
print(f"Executing: {build_command} with environment: {build_context.container_build_env}")
build_result = subprocess.run(build_command, shell=True, env=build_context.container_build_env)
if opts.o.verbose:
print(f"Return code is: {build_result.returncode}")
if build_result.returncode != 0:
print(f"Error running build for {container}")
if not continue_on_error:
print("FATAL Error: container build failed and --continue-on-error not set, exiting")
sys.exit(1)
else:
print("****** Container Build Error, continuing because --continue-on-error is set")
return False
else:
return True
else:
print("Skipped")
return True
@click.command()
@@ -129,17 +121,14 @@ def process_container(stack: str,
@click.option('--exclude', help="don\'t build these containers")
@click.option("--force-rebuild", is_flag=True, default=False, help="Override dependency checking -- always rebuild")
@click.option("--extra-build-args", help="Supply extra arguments to build")
@click.option("--publish-images", is_flag=True, default=False, help="Publish the built images in the specified image registry")
@click.option("--image-registry", help="Specify the image registry for --publish-images")
@click.pass_context
def command(ctx, include, exclude, force_rebuild, extra_build_args):
def command(ctx, include, exclude, force_rebuild, extra_build_args, publish_images, image_registry):
'''build the set of containers required for a complete stack'''
quiet = ctx.obj.quiet
verbose = ctx.obj.verbose
dry_run = ctx.obj.dry_run
debug = ctx.obj.debug
local_stack = ctx.obj.local_stack
stack = ctx.obj.stack
continue_on_error = ctx.obj.continue_on_error
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
container_build_dir = Path(__file__).absolute().parent.parent.joinpath("data", "container-build")
@@ -150,41 +139,45 @@ def command(ctx, include, exclude, force_rebuild, extra_build_args):
else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
if not quiet:
if not opts.o.quiet:
print(f'Dev Root is: {dev_root_path}')
if not os.path.isdir(dev_root_path):
print('Dev root directory doesn\'t exist, creating')
# See: https://stackoverflow.com/a/20885799/1701505
from stack_orchestrator import data
with importlib.resources.open_text(data, "container-image-list.txt") as container_list_file:
all_containers = container_list_file.read().splitlines()
if publish_images:
if not image_registry:
error_exit("--image-registry must be supplied with --publish-images")
containers_in_scope = []
if stack:
stack_config = get_parsed_stack_config(stack)
if "containers" not in stack_config or stack_config["containers"] is None:
warn_exit(f"stack {stack} does not define any containers")
containers_in_scope = stack_config['containers']
else:
containers_in_scope = all_containers
if verbose:
print(f'Containers: {containers_in_scope}')
if stack:
print(f"Stack: {stack}")
containers_in_scope = get_containers_in_scope(stack)
container_build_env = make_container_build_env(dev_root_path,
container_build_dir,
debug,
opts.o.debug,
force_rebuild,
extra_build_args)
for container in containers_in_scope:
if include_exclude_check(container, include, exclude):
process_container(stack, container, container_build_dir, container_build_env,
dev_root_path, quiet, verbose, dry_run, continue_on_error)
build_context = BuildContext(
stack,
container,
container_build_dir,
container_build_env,
dev_root_path
)
result = process_container(build_context)
if result:
if publish_images:
publish_image(f"{container}:local", image_registry)
else:
print(f"Error running build for {build_context.container}")
if not opts.o.continue_on_error:
error_exit("container build failed and --continue-on-error not set, exiting")
sys.exit(1)
else:
print("****** Container Build Error, continuing because --continue-on-error is set")
else:
if verbose:
if opts.o.verbose:
print(f"Excluding: {container}")
+29
View File
@@ -0,0 +1,29 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
@dataclass
class BuildContext:
stack: str
container: str
container_build_dir: Path
container_build_env: Mapping[str,str]
dev_root_path: str
+43
View File
@@ -0,0 +1,43 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import importlib.resources
from stack_orchestrator.opts import opts
from stack_orchestrator.util import get_parsed_stack_config, warn_exit
def get_containers_in_scope(stack: str):
# See: https://stackoverflow.com/a/20885799/1701505
from stack_orchestrator import data
with importlib.resources.open_text(data, "container-image-list.txt") as container_list_file:
all_containers = container_list_file.read().splitlines()
containers_in_scope = []
if stack:
stack_config = get_parsed_stack_config(stack)
if "containers" not in stack_config or stack_config["containers"] is None:
warn_exit(f"stack {stack} does not define any containers")
containers_in_scope = stack_config['containers']
else:
containers_in_scope = all_containers
if opts.o.verbose:
print(f'Containers: {containers_in_scope}')
if stack:
print(f"Stack: {stack}")
return containers_in_scope
+47 -14
View File
@@ -21,11 +21,14 @@
# TODO: display the available list of containers; allow re-build of either all or specific containers
import os
import sys
from decouple import config
import click
from pathlib import Path
from stack_orchestrator.build import build_containers
from stack_orchestrator.deploy.webapp.util import determine_base_container
from stack_orchestrator.deploy.webapp.util import determine_base_container, TimedLogger
from stack_orchestrator.build.build_types import BuildContext
@click.command()
@@ -37,26 +40,25 @@ from stack_orchestrator.deploy.webapp.util import determine_base_container
@click.pass_context
def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, tag):
'''build the specified webapp container'''
logger = TimedLogger()
quiet = ctx.obj.quiet
verbose = ctx.obj.verbose
dry_run = ctx.obj.dry_run
debug = ctx.obj.debug
verbose = ctx.obj.verbose
local_stack = ctx.obj.local_stack
stack = ctx.obj.stack
continue_on_error = ctx.obj.continue_on_error
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
container_build_dir = Path(__file__).absolute().parent.parent.joinpath("data", "container-build")
if local_stack:
dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
logger.log(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
if not quiet:
print(f'Dev Root is: {dev_root_path}')
if verbose:
logger.log(f'Dev Root is: {dev_root_path}')
if not base_container:
base_container = determine_base_container(source_repo)
@@ -65,8 +67,23 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
container_build_env = build_containers.make_container_build_env(dev_root_path, container_build_dir, debug,
force_rebuild, extra_build_args)
build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
verbose, dry_run, continue_on_error)
if verbose:
logger.log(f"Building base container: {base_container}")
build_context_1 = BuildContext(
stack,
base_container,
container_build_dir,
container_build_env,
dev_root_path,
)
ok = build_containers.process_container(build_context_1)
if not ok:
logger.log("ERROR: Build failed.")
sys.exit(1)
if verbose:
logger.log(f"Base container {base_container} build finished.")
# Now build the target webapp. We use the same build script, but with a different Dockerfile and work dir.
container_build_env["CERC_WEBAPP_BUILD_RUNNING"] = "true"
@@ -76,9 +93,25 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
"Dockerfile.webapp")
if not tag:
webapp_name = os.path.abspath(source_repo).split(os.path.sep)[-1]
container_build_env["CERC_CONTAINER_BUILD_TAG"] = f"cerc/{webapp_name}:local"
else:
container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
tag = f"cerc/{webapp_name}:local"
build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
verbose, dry_run, continue_on_error)
container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
if verbose:
logger.log(f"Building app container: {tag}")
build_context_2 = BuildContext(
stack,
base_container,
container_build_dir,
container_build_env,
dev_root_path,
)
ok = build_containers.process_container(build_context_2)
if not ok:
logger.log("ERROR: Build failed.")
sys.exit(1)
if verbose:
logger.log(f"App container {base_container} build finished.")
logger.log("build-webapp complete", show_step_time=False, show_total_time=True)
@@ -0,0 +1,195 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import click
from dataclasses import dataclass
import json
import platform
from python_on_whales import DockerClient
from python_on_whales.components.manifest.cli_wrapper import ManifestCLI, ManifestList
from python_on_whales.utils import run
import requests
from typing import List
from stack_orchestrator.opts import opts
from stack_orchestrator.util import include_exclude_check, error_exit
from stack_orchestrator.build.build_util import get_containers_in_scope
# Experimental fetch-container command
@dataclass
class RegistryInfo:
registry: str
registry_username: str
registry_token: str
# Extending this code to support the --verbose option, cnosider contributing upstream
# https://github.com/gabrieldemarmiesse/python-on-whales/blob/master/python_on_whales/components/manifest/cli_wrapper.py#L129
class ExtendedManifestCLI(ManifestCLI):
def inspect_verbose(self, x: str) -> ManifestList:
"""Returns a Docker manifest list object."""
json_str = run(self.docker_cmd + ["manifest", "inspect", "--verbose", x])
return json.loads(json_str)
def _local_tag_for(container: str):
return f"{container}:local"
# See: https://docker-docs.uclv.cu/registry/spec/api/
# Emulate this:
# $ curl -u "my-username:my-token" -X GET "https://<container-registry-hostname>/v2/cerc-io/cerc/test-container/tags/list"
# {"name":"cerc-io/cerc/test-container","tags":["202402232130","202402232208"]}
def _get_tags_for_container(container: str, registry_info: RegistryInfo) -> List[str]:
# registry looks like: git.vdb.to/cerc-io
registry_parts = registry_info.registry.split("/")
url = f"https://{registry_parts[0]}/v2/{registry_parts[1]}/{container}/tags/list"
if opts.o.debug:
print(f"Fetching tags from: {url}")
response = requests.get(url, auth=(registry_info.registry_username, registry_info.registry_token))
if response.status_code == 200:
tag_info = response.json()
if opts.o.debug:
print(f"container tags list: {tag_info}")
tags_array = tag_info["tags"]
return tags_array
else:
error_exit(f"failed to fetch tags from image registry, status code: {response.status_code}")
def _find_latest(candidate_tags: List[str]):
# Lex sort should give us the latest first
sorted_candidates = sorted(candidate_tags)
if opts.o.debug:
print(f"sorted candidates: {sorted_candidates}")
return sorted_candidates[-1]
def _filter_for_platform(container: str,
registry_info: RegistryInfo,
tag_list: List[str]) -> List[str] :
filtered_tags = []
this_machine = platform.machine()
# Translate between Python and docker platform names
if this_machine == "x86_64":
this_machine = "amd64"
if this_machine == "aarch64":
this_machine = "arm64"
if opts.o.debug:
print(f"Python says the architecture is: {this_machine}")
docker = DockerClient()
for tag in tag_list:
remote_tag = f"{registry_info.registry}/{container}:{tag}"
manifest_cmd = ExtendedManifestCLI(docker.client_config)
manifest = manifest_cmd.inspect_verbose(remote_tag)
if opts.o.debug:
print(f"manifest: {manifest}")
image_architecture = manifest["Descriptor"]["platform"]["architecture"]
if opts.o.debug:
print(f"image_architecture: {image_architecture}")
if this_machine == image_architecture:
filtered_tags.append(tag)
if opts.o.debug:
print(f"Tags filtered for platform: {filtered_tags}")
return filtered_tags
def _get_latest_image(container: str, registry_info: RegistryInfo):
all_tags = _get_tags_for_container(container, registry_info)
tags_for_platform = _filter_for_platform(container, registry_info, all_tags)
if len(tags_for_platform) > 0:
latest_tag = _find_latest(tags_for_platform)
return f"{container}:{latest_tag}"
else:
return None
def _fetch_image(tag: str, registry_info: RegistryInfo):
docker = DockerClient()
remote_tag = f"{registry_info.registry}/{tag}"
if opts.o.debug:
print(f"Attempting to pull this image: {remote_tag}")
docker.image.pull(remote_tag)
def _exists_locally(container: str):
docker = DockerClient()
return docker.image.exists(_local_tag_for(container))
def _add_local_tag(remote_tag: str, registry: str, local_tag: str):
docker = DockerClient()
docker.image.tag(f"{registry}/{remote_tag}", local_tag)
@click.command()
@click.option('--include', help="only fetch these containers")
@click.option('--exclude', help="don\'t fetch these containers")
@click.option("--force-local-overwrite", is_flag=True, default=False, help="Overwrite a locally built image, if present")
@click.option("--image-registry", required=True, help="Specify the image registry to fetch from")
@click.option("--registry-username", required=True, help="Specify the image registry username")
@click.option("--registry-token", required=True, help="Specify the image registry access token")
@click.pass_context
def command(ctx, include, exclude, force_local_overwrite, image_registry, registry_username, registry_token):
'''EXPERIMENTAL: fetch the images for a stack from remote registry'''
registry_info = RegistryInfo(image_registry, registry_username, registry_token)
docker = DockerClient()
if not opts.o.quiet:
print("Logging into container registry:")
docker.login(registry_info.registry, registry_info.registry_username, registry_info.registry_token)
# Generate list of target containers
stack = ctx.obj.stack
containers_in_scope = get_containers_in_scope(stack)
all_containers_found = True
for container in containers_in_scope:
local_tag = _local_tag_for(container)
if include_exclude_check(container, include, exclude):
if opts.o.debug:
print(f"Processing: {container}")
# For each container, attempt to find the latest of a set of
# images with the correct name and platform in the specified registry
image_to_fetch = _get_latest_image(container, registry_info)
if not image_to_fetch:
print(f"Warning: no image found to fetch for container: {container}")
all_containers_found = False
continue
if opts.o.debug:
print(f"Fetching: {image_to_fetch}")
_fetch_image(image_to_fetch, registry_info)
# Now check if the target container already exists exists locally already
if (_exists_locally(container)):
if not opts.o.quiet:
print(f"Container image {container} already exists locally")
# if so, fail unless the user specified force-local-overwrite
if (force_local_overwrite):
# In that case remove the existing :local tag
if not opts.o.quiet:
print(f"Warning: overwriting local tag from this image: {container} because "
"--force-local-overwrite was specified")
else:
if not opts.o.quiet:
print(f"Skipping local tagging for this image: {container} because that would "
"overwrite an existing :local tagged image, use --force-local-overwrite to do so.")
continue
# Tag the fetched image with the :local tag
_add_local_tag(image_to_fetch, image_registry, local_tag)
else:
if opts.o.verbose:
print(f"Excluding: {container}")
if not all_containers_found:
print("Warning: couldn't find usable images for one or more containers, this stack will not deploy")
+48
View File
@@ -0,0 +1,48 @@
# Copyright © 2024 Vulcanize
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
from datetime import datetime
from python_on_whales import DockerClient
from stack_orchestrator.opts import opts
from stack_orchestrator.util import error_exit
def _publish_tag_for_image(local_image_tag: str, remote_repo: str, version: str):
# Turns image tags of the form: foo/bar:local into remote.repo/org/bar:deploy
(image_name, image_version) = local_image_tag.split(":")
if image_version == "local":
return f"{remote_repo}/{image_name}:{version}"
else:
error_exit("Asked to publish a non-locally built image")
def publish_image(local_tag, registry):
if opts.o.verbose:
print(f"Publishing this image: {local_tag} to this registry: {registry}")
docker = DockerClient()
# Figure out the target image tag
# Eventually this version will be generated from the source repo state
# Using a timestemp is an intermediate step
version = datetime.now().strftime("%Y%m%d%H%M")
remote_tag = _publish_tag_for_image(local_tag, registry, version)
# Tag the image thus
if opts.o.debug:
print(f"Tagging {local_tag} to {remote_tag}")
docker.image.tag(local_tag, remote_tag)
# Push it to the desired registry
if opts.o.verbose:
print(f"Pushing image {remote_tag}")
docker.image.push(remote_tag)
@@ -5,9 +5,11 @@ services:
environment:
REGISTRY_LOG_LEVEL: ${REGISTRY_LOG_LEVEL}
volumes:
- config:/config:ro
- registry-data:/var/lib/registry
ports:
- "5000"
volumes:
config:
registry-data:
@@ -0,0 +1,141 @@
version: '3.7'
services:
go-nitro-alice:
image: cerc/go-nitro:local
hostname: go-nitro-alice
restart: on-failure
depends_on:
- go-nitro-bootnode
environment:
CERC_NITRO_CHAIN_PK: ${CERC_NITRO_CHAIN_PK_ALICE:-888814df89c4358d7ddb3fa4b0213e7331239a80e1f013eaa7b2deca2a41a218}
CERC_NITRO_CHAIN_URL: ${CERC_NITRO_CHAIN_URL:-ws://fixturenet-eth-geth-1:8546}
CERC_NA_ADDRESS: ${CERC_NA_ADDRESS}
CERC_VPA_ADDRESS: ${CERC_VPA_ADDRESS}
CERC_CA_ADDRESS: ${CERC_CA_ADDRESS}
env_file:
- ../config/nitro-auth/alice.env
entrypoint: ["bash", "-c", "/app/run-nitro-node.sh"]
volumes:
- nitro_deployment:/app/deployment
- go_nitro_data_alice:/app/data
- ../config/go-nitro/run-nitro-node.sh:/app/run-nitro-node.sh
healthcheck:
test: ["CMD", "nc", "-vz", "localhost", "4006"]
interval: 5s
timeout: 5s
retries: 100
start_period: 100s
ports:
- "3006"
- "4006"
- "5006"
- "4106"
- "4206"
- "4216"
go-nitro-bob:
image: cerc/go-nitro:local
hostname: go-nitro-bob
restart: on-failure
depends_on:
- go-nitro-bootnode
environment:
CERC_NITRO_CHAIN_PK: ${CERC_NITRO_CHAIN_PK_BOB:-570b909da9669b2f35a0b1ac70b8358516d55ae1b5b3710e95e9a94395090597}
CERC_NITRO_CHAIN_URL: ${CERC_NITRO_CHAIN_URL:-ws://fixturenet-eth-geth-1:8546}
CERC_NA_ADDRESS: ${CERC_NA_ADDRESS}
CERC_VPA_ADDRESS: ${CERC_VPA_ADDRESS}
CERC_CA_ADDRESS: ${CERC_CA_ADDRESS}
env_file:
- ../config/nitro-auth/bob.env
entrypoint: ["bash", "-c", "/app/run-nitro-node.sh"]
volumes:
- nitro_deployment:/app/deployment
- go_nitro_data_bob:/app/data
- ../config/go-nitro/run-nitro-node.sh:/app/run-nitro-node.sh
healthcheck:
test: ["CMD", "nc", "-vz", "localhost", "4007"]
interval: 5s
timeout: 5s
retries: 100
start_period: 100s
ports:
- "3007"
- "4007"
- "5007"
- "4107"
- "4207"
- "4217"
ts-nitro-charlie:
image: cerc/ts-nitro:local
hostname: ts-nitro-charlie
restart: on-failure
depends_on:
- go-nitro-bootnode
environment:
CERC_NITRO_CHAIN_PK: ${CERC_NITRO_CHAIN_PK_CHARLIE:-111b7500bdce494d6f4bcfe8c2a0dde2ef92f751d9070fac6475dbd6d8021b3f}
CERC_NITRO_CHAIN_URL: ${CERC_NITRO_CHAIN_URL:-ws://fixturenet-eth-geth-1:8546}
CERC_NA_ADDRESS: ${CERC_NA_ADDRESS}
CERC_VPA_ADDRESS: ${CERC_VPA_ADDRESS}
CERC_CA_ADDRESS: ${CERC_CA_ADDRESS}
CERC_NITRO_TARGET_URL: ${CERC_NITRO_TARGET_URL:-http://localhost:5678}
CERC_RUNTIME_ENV_RPC_URL: ${CERC_RUNTIME_ENV_RPC_URL:-ws://localhost:8546}
PUBLIC_URL: http://localhost:5678/ts-nitro
env_file:
- ../config/nitro-auth/charlie.env
entrypoint: ["bash", "-c", "/app/run.sh"]
volumes:
- nitro_deployment:/app/deployment
- ../config/ts-nitro/run.sh:/app/run.sh
healthcheck:
test: ["CMD", "nc", "-vz", "localhost", "3000"]
interval: 5s
timeout: 5s
retries: 100
start_period: 100s
go-nitro-bootnode:
image: cerc/go-nitro:local
hostname: go-nitro-bootnode
restart: on-failure
environment:
CERC_NITRO_CHAIN_URL: ${CERC_NITRO_CHAIN_URL:-ws://fixturenet-eth-geth-1:8546}
CERC_NA_ADDRESS: ${CERC_NA_ADDRESS}
CERC_VPA_ADDRESS: ${CERC_VPA_ADDRESS}
CERC_CA_ADDRESS: ${CERC_CA_ADDRESS}
CERC_NITRO_RELAY_ON: true
env_file:
- ../config/nitro-auth/bootnode.env
entrypoint: [ "bash", "-c", "/app/run-nitro-node.sh" ]
volumes:
- nitro_deployment:/app/deployment
- go_nitro_data_bootnode:/app/data
- ../config/go-nitro/run-nitro-node.sh:/app/run-nitro-node.sh
healthcheck:
test: [ "CMD", "nc", "-vz", "localhost", "4008" ]
interval: 5s
timeout: 5s
retries: 100
start_period: 100s
ports:
- "9090"
nitro-caddy:
image: caddy:2-alpine
restart: always
environment:
CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT:-http://fixturenet-eth-geth-1:8545}
volumes:
- ../config/nitro-auth/caddy/Caddyfile:/etc/caddy/Caddyfile
ports:
- "5678"
depends_on:
- go-nitro-alice
- go-nitro-bob
volumes:
go_nitro_data_alice:
go_nitro_data_bob:
go_nitro_data_bootnode:
nitro_deployment:
@@ -4,6 +4,11 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
if [ -z "$CERC_NITRO_CHAIN_PK" ] || [ -z "$CERC_NITRO_CHAIN_URL" ]; then
echo "You most set both CERC_NITRO_CHAIN_PK and CERC_NITRO_CHAIN_URL." 1>&2
exit 1
fi
nitro_addresses_file="/app/deployment/nitro-addresses.json"
# Check if CERC_NA_ADDRESS environment variable is set
@@ -29,22 +34,86 @@ fi
echo "Running Nitro node"
# Assuming CERC_NITRO_CHAIN_URL is of format <ws|http>://host:port
ws_host=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d ':' -f 1)
ws_port=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d ':' -f 2)
# Wait till chain endpoint is available
retry_interval=5
while true; do
nc -z -w 1 "$ws_host" "$ws_port"
if [ $? -eq 0 ]; then
echo "Chain endpoint is available"
break
if [[ "${CERC_GO_NITRO_WAIT_FOR_CHAIN:-true}" == "true" ]]; then
# Assuming CERC_NITRO_CHAIN_URL is of format <ws|http>://host[:port][/foo]
ws_host=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d'/' -f 1 | cut -d ':' -f 1)
ws_port=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d'/' -f 1 | cut -d ':' -f 2)
if [[ "$ws_port" == "$ws_host" ]]; then
ws_port=""
fi
echo "Chain endpoint not yet available, retrying in $retry_interval seconds..."
sleep $retry_interval
done
# Wait till chain endpoint is available
retry_interval=5
while true; do
nc -z -w 1 "$ws_host" "${ws_port:-443}"
./nitro -chainurl ${CERC_NITRO_CHAIN_URL} -msgport ${CERC_NITRO_MSG_PORT} -rpcport ${CERC_NITRO_RPC_PORT} -wsmsgport ${CERC_NITRO_WS_MSG_PORT} -publicip "0.0.0.0" -pk ${CERC_NITRO_PK} -chainpk ${CERC_NITRO_CHAIN_PK} -naaddress ${NA_ADDRESS} -vpaaddress ${VPA_ADDRESS} -caaddress ${CA_ADDRESS} -usedurablestore ${CERC_NITRO_USE_DURABLE_STORE} -durablestorefolder ${CERC_NITRO_DURABLE_STORE_FOLDER} -tlscertfilepath "" -tlsKeyFilepath ""
if [ $? -eq 0 ]; then
echo "Chain endpoint is available"
break
fi
echo "Chain endpoint not yet available, retrying in $retry_interval seconds..."
sleep $retry_interval
done
fi
if [[ -n "$CERC_NITRO_UI_PORT" ]] && [[ -d "/app-node/packages/nitro-gui/dist" ]]; then
for f in `ls /app-node/packages/nitro-gui/dist/assets/*.js`; do
sed -i "s#\"CERC_RUNTIME_ENV_RPC_HOST\"#\"localhost:${CERC_NITRO_RPC_PORT}\"#g" "$f"
sed -i "s#\"CERC_RUNTIME_ENV_TARGET_URL\"#\"http://localhost:5678\"#g" "$f"
done
http-server -p $CERC_NITRO_UI_PORT /app-node/packages/nitro-gui/dist &
fi
if [[ -n "$CERC_NITRO_AUTH_UI_PORT" ]] && [[ -d "/app-node/packages/nitro-auth-gui/dist" ]]; then
for f in `ls /app-node/packages/nitro-auth-gui/dist/assets/*.js`; do
sed -i "s#\"CERC_RUNTIME_ENV_RPC_URL\"#\"http://localhost:${CERC_NITRO_RPC_PORT}\"#g" "$f"
sed -i "s#\"CERC_RUNTIME_ENV_TARGET_URL\"#\"http://localhost:5678\"#g" "$f"
done
http-server -p $CERC_NITRO_AUTH_UI_PORT /app-node/packages/nitro-auth-gui/dist &
fi
if [[ "$CERC_NITRO_AUTH_ON" == "true" ]] && [[ -d "/app-node/packages/nitro-auth/dist" ]]; then
bash -c "sleep 6 && cd /app-node/packages/nitro-auth && yarn start" &
fi
if [[ "$CERC_NITRO_RELAY_ON" == "true" ]]; then
if [[ ! -f "/app/deployment/relay-node.json" ]]; then
node /usr/local/lib/node_modules/@cerc-io/peer/dist/cli/create-peer.js \
-f /app/deployment/relay-node.json
fi
DEBUG='laconic:*' node /usr/local/lib/node_modules/@cerc-io/peer/dist/cli/relay.js \
--host 0.0.0.0 \
-f /app/deployment/relay-node.json &
fi
if [[ -z "$CERC_CHAIN_START_BLOCK" ]]; then
if [[ ! -f "/app/deployment/chainstartblock.json" ]]; then
curl --location "$(echo $CERC_NITRO_CHAIN_URL | sed 's/^ws/http/' | sed 's#/ws/#/#')" \
--header 'Content-Type: application/json' \
--data '{
"jsonrpc": "2.0",
"id": 124,
"method": "eth_blockNumber",
"params": []
}' > /app/deployment/chainstartblock.json
fi
CERC_CHAIN_START_BLOCK=$(printf "%d" `cat /app/deployment/chainstartblock.json | jq -r '.result'`)
fi
cd /app
./nitro \
-chainurl ${CERC_NITRO_CHAIN_URL} \
-msgport ${CERC_NITRO_MSG_PORT} \
-rpcport ${CERC_NITRO_RPC_PORT} \
-wsmsgport ${CERC_NITRO_WS_MSG_PORT} \
-publicip "0.0.0.0" \
-pk ${CERC_NITRO_PK:-$CERC_NITRO_CHAIN_PK} \
-chainpk ${CERC_NITRO_CHAIN_PK} \
-naaddress ${NA_ADDRESS} \
-vpaaddress ${VPA_ADDRESS} \
-caaddress ${CA_ADDRESS} \
-usedurablestore=${CERC_NITRO_USE_DURABLE_STORE} \
-durablestorefolder ${CERC_NITRO_DURABLE_STORE_FOLDER} \
-bootpeers "${CERC_NITRO_BOOT_PEERS}" \
-chainstartblock $CERC_CHAIN_START_BLOCK
@@ -0,0 +1,19 @@
# SET ME! Your on-chain private key.
#CERC_NITRO_CHAIN_PK=<MY_PK_GOES_HERE>
# Default is CERC_NITRO_CHAIN_PK.
#CERC_NITRO_PK=<MY_PK_GOES_HERE>
# SET ME! The WebSocket to connect to.
#CERC_NITRO_CHAIN_URL=wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY_GOES_HERE>
CERC_NITRO_AUTH_ON=false
CERC_NITRO_AUTH_RPC_PORT=4006
CERC_NITRO_AUTH_UI_PORT=4206
CERC_NITRO_BOOT_PEERS="/dns4/go-nitro-bootnode/tcp/3008/p2p/16Uiu2HAmFYz5YPf3GtkfQsAwYJfDsKSm34U9qAiTwxW7RJUVGMRK"
CERC_NITRO_DURABLE_STORE_FOLDER=/app/data/nitro-store
CERC_NITRO_MSG_PORT=3006
CERC_NITRO_RPC_PORT=4006
CERC_NITRO_UI_PORT=4106
CERC_NITRO_USE_DURABLE_STORE=true
CERC_NITRO_WS_MSG_PORT=5006
@@ -0,0 +1,21 @@
# SET ME! Your on-chain private key.
#CERC_NITRO_CHAIN_PK=<MY_PK_GOES_HERE>
# Default is CERC_NITRO_CHAIN_PK.
#CERC_NITRO_PK=<MY_PK_GOES_HERE>
# SET ME! The WebSocket to connect to.
#CERC_NITRO_CHAIN_URL=wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY_GOES_HERE>
CERC_NITRO_AUTH_ON=true
CERC_NITRO_AUTH_RPC_PORT=4007
CERC_NITRO_AUTH_UI_PORT=4207
CERC_NITRO_BOOT_PEERS="/dns4/go-nitro-bootnode/tcp/3008/p2p/16Uiu2HAmFYz5YPf3GtkfQsAwYJfDsKSm34U9qAiTwxW7RJUVGMRK"
CERC_NITRO_DURABLE_STORE_FOLDER=/app/data/nitro-store
CERC_NITRO_MSG_PORT=3007
CERC_NITRO_RPC_PORT=4007
CERC_NITRO_UI_PORT=4107
CERC_NITRO_USE_DURABLE_STORE=true
CERC_NITRO_WS_MSG_PORT=5007
CERC_NITRO_MSG_PUBLIC_IP=127.0.0.1
CERC_NITRO_MSG_PUBLIC_PORT=5007
@@ -0,0 +1,17 @@
CERC_NITRO_CHAIN_PK=596db2ac27479cfdf60f708bf64ae44a1c5090e9446cca011d72bc9a59b47d3d
CERC_NITRO_PK=f41086394674cf00a66448c6688295d7330af39f6f38fed89ea023e39382d6a0
# The WebSocket to connect to.
#CERC_NITRO_CHAIN_URL=wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY_GOES_HERE>
CERC_NITRO_AUTH_ON=false
CERC_NITRO_AUTH_RPC_PORT=4008
CERC_NITRO_AUTH_UI_PORT=4208
CERC_NITRO_DURABLE_STORE_FOLDER=/app/data/nitro-store
CERC_NITRO_MSG_PORT=3008
CERC_NITRO_RPC_PORT=4008
CERC_NITRO_UI_PORT=4108
CERC_NITRO_USE_DURABLE_STORE=true
CERC_NITRO_WS_MSG_PORT=5008
CERC_SCRIPT_DEBUG=true
@@ -0,0 +1,48 @@
:5678 {
handle /pay* {
reverse_proxy http://go-nitro-bob:8547
}
@eth-key-in-header {
path_regexp /eth/?$
header X-Api-Key *
}
handle @eth-key-in-header {
forward_auth go-nitro-bob:8547 {
uri /auth/{header.x-api-key}
}
rewrite * /
reverse_proxy {$CERC_ETH_RPC_ENDPOINT} {
header_up Host {upstream_hostport}
header_up -X-Forwarded-*
}
}
@eth-key-in-path {
path_regexp apikey eth/(.+)$
}
handle @eth-key-in-path {
forward_auth go-nitro-bob:8547 {
uri /auth/{re.apikey.1}
}
rewrite * /
reverse_proxy {$CERC_ETH_RPC_ENDPOINT} {
header_up Host {upstream_hostport}
header_up -X-Forwarded-*
}
}
handle /eth* {
respond "401 Unauthorized" 401
}
handle /ts-nitro* {
reverse_proxy http://ts-nitro-charlie:3000
}
handle {
reverse_proxy http://go-nitro-alice:4206
}
}
@@ -0,0 +1,19 @@
# SET ME! Your on-chain private key.
#CERC_NITRO_CHAIN_PK=<MY_PK_GOES_HERE>
# Default is CERC_NITRO_CHAIN_PK.
#CERC_NITRO_PK=<MY_PK_GOES_HERE>
# SET ME! The WebSocket to connect to.
#CERC_NITRO_CHAIN_URL=wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY_GOES_HERE>
CERC_NITRO_AUTH_ON=false
CERC_NITRO_AUTH_RPC_PORT=4009
CERC_NITRO_AUTH_UI_PORT=4209
CERC_NITRO_BOOT_PEERS="/dns4/go-nitro-bootnode/tcp/3008/p2p/16Uiu2HAmFYz5YPf3GtkfQsAwYJfDsKSm34U9qAiTwxW7RJUVGMRK"
CERC_NITRO_DURABLE_STORE_FOLDER=/app/data/nitro-store
CERC_NITRO_MSG_PORT=3009
CERC_NITRO_RPC_PORT=4009
CERC_NITRO_UI_PORT=4109
CERC_NITRO_USE_DURABLE_STORE=true
CERC_NITRO_WS_MSG_PORT=5009
+99
View File
@@ -0,0 +1,99 @@
#!/bin/bash
CERC_NITRO_RPC_FUND_AMOUNT=${CERC_NITRO_RPC_FUND_AMOUNT:-0}
CERC_NITRO_RPC_HOST_ALICE=${CERC_NITRO_RPC_HOST_ALICE:-go-nitro-alice}
CERC_NITRO_RPC_PORT_ALICE=${CERC_NITRO_RPC_PORT_ALICE:-4006}
CERC_NITRO_USE_TLS=${CERC_NITRO_USE_TLS:-false}
CERC_NITRO_ADDRESS_BOB=${CERC_NITRO_ADDRESS_BOB:-0xe07e314501cc73b24cf45a6577486017300e153c}
# Wait till chain endpoint is available
retry_interval=5
while true; do
nc -z -w 1 "$CERC_NITRO_RPC_HOST_ALICE" "$CERC_NITRO_RPC_PORT_ALICE"
if [ $? -eq 0 ]; then
echo "Nitro endpoint is available"
break
fi
echo "Nitro endpoint not yet available, retrying in $retry_interval seconds..."
sleep $retry_interval
done
if [[ "$CERC_NITRO_RPC_FUND_AMOUNT" -gt 0 ]]; then
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
get-all-ledger-channels | \
jq "[.[] | select(.Status == \"Open\") | select(.Balance.Them == \"$CERC_NITRO_ADDRESS_BOB\")] | first" > \
/app/deployment/nitro-ledger-channel-alice-to-bob.json
ledger_channel=$(jq -r '.ID' /app/deployment/nitro-ledger-channel-alice-to-bob.json 2>/dev/null | sed 's/^null$//')
if [[ -z "${ledger_channel}" ]]; then
echo "Creating new ledger channel ..."
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
-n \
direct-fund --amount $CERC_NITRO_RPC_FUND_AMOUNT $CERC_NITRO_ADDRESS_BOB
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
get-all-ledger-channels | \
jq "[.[] | select(.Status == \"Open\") | select(.Balance.Them == \"$CERC_NITRO_ADDRESS_BOB\")] | first" > \
/app/deployment/nitro-ledger-channel-alice-to-bob.json
ledger_channel=$(jq -r '.ID' /app/deployment/nitro-ledger-channel-alice-to-bob.json)
fi
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
get-payment-channels-by-ledger $ledger_channel > \
/app/deployment/nitro-payment-channels-alice-to-bob.json
first_open_channel=$(jq '[.[] | select(.Status == "Open")] | first' /app/deployment/nitro-payment-channels-alice-to-bob.json | sed 's/^null$//')
if [[ -z "$first_open_channel" ]]; then
echo "Creating new payment channel ..."
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
-n \
virtual-fund --amount $((CERC_NITRO_RPC_FUND_AMOUNT/2)) $CERC_NITRO_ADDRESS_BOB
nitro-rpc-client -h $CERC_NITRO_RPC_HOST_ALICE \
-p $CERC_NITRO_RPC_PORT_ALICE \
-s=$CERC_NITRO_USE_TLS \
get-payment-channels-by-ledger $ledger_channel > \
/app/deployment/nitro-payment-channels-alice-to-bob.json
first_open_channel=$(jq '[.[] | select(.Status == "Open")] | first' /app/deployment/nitro-payment-channels-alice-to-bob.json | sed 's/^null$//')
fi
echo ""
echo "################################################################"
echo ""
echo "LEDGER:"
cat /app/deployment/nitro-ledger-channel-alice-to-bob.json | jq
echo ""
echo ""
echo "PAYMENT:"
cat /app/deployment/nitro-payment-channels-alice-to-bob.json | jq
echo ""
echo ""
fi
if [ -n "$1" ]; then
exec "$@"
exit $?
fi
while [ 1 -eq 1 ]; do
sleep 100
done
+45
View File
@@ -0,0 +1,45 @@
#!/bin/bash
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
if [ -z "$CERC_NITRO_CHAIN_PK" ] || [ -z "$CERC_NITRO_CHAIN_URL" ]; then
echo "You most set both CERC_NITRO_CHAIN_PK and CERC_NITRO_CHAIN_URL." 1>&2
exit 1
fi
nitro_addresses_file="/app/deployment/nitro-addresses.json"
# Check if CERC_NA_ADDRESS environment variable is set
if [ -n "$CERC_NA_ADDRESS" ]; then
echo "CERC_NA_ADDRESS is set to '$CERC_NA_ADDRESS'"
echo "CERC_VPA_ADDRESS is set to '$CERC_VPA_ADDRESS'"
echo "CERC_CA_ADDRESS is set to '$CERC_CA_ADDRESS'"
echo "Using the above Nitro addresses"
NA_ADDRESS=${CERC_NA_ADDRESS}
VPA_ADDRESS=${CERC_VPA_ADDRESS}
CA_ADDRESS=${CERC_CA_ADDRESS}
elif [ -f ${nitro_addresses_file} ]; then
echo "Reading Nitro addresses from ${nitro_addresses_file}"
NA_ADDRESS=$(jq -r '.nitroAdjudicatorAddress' ${nitro_addresses_file})
VPA_ADDRESS=$(jq -r '.virtualPaymentAppAddress' ${nitro_addresses_file})
CA_ADDRESS=$(jq -r '.consensusAppAddress' ${nitro_addresses_file})
else
echo "File ${nitro_addresses_file} not found"
exit 1
fi
cd /app/packages/example-web-app
cat > .env <<EOF
REACT_APP_RPC_URL=${CERC_RUNTIME_ENV_RPC_URL:-$CERC_NITRO_CHAIN_URL}
REACT_APP_TARGET_URL=${CERC_RUNTIME_ENV_TARGET_URL:-$CERC_NITRO_TARGET_URL}
REACT_APP_NITRO_PK=${CERC_NITRO_PK:-$CERC_NITRO_CHAIN_PK}
REACT_APP_NA_ADDRESS=${NA_ADDRESS}
REACT_APP_VPA_ADDRESS=${VPA_ADDRESS}
REACT_APP_CA_ADDRESS=${CA_ADDRESS}
EOF
yarn start
@@ -6,7 +6,7 @@ fi
ETHERBASE=`cat /opt/testnet/build/el/accounts.csv | head -1 | cut -d',' -f2`
NETWORK_ID=`cat /opt/testnet/el/el-config.yaml | grep 'chain_id' | awk '{ print $2 }'`
NETRESTRICT=`ip addr | grep inet | grep -v '127.0' | awk '{print $2}'`
NETRESTRICT=`ip addr | grep 'inet ' | grep -v '127.0' | head -1 | awk '{print $2}'`
CERC_ETH_DATADIR="${CERC_ETH_DATADIR:-$HOME/ethdata}"
CERC_PLUGINS_DIR="${CERC_PLUGINS_DIR:-/usr/local/lib/plugeth}"
@@ -8,9 +8,30 @@ COPY . .
RUN go build -v -o nitro .
# Reduce image size
FROM debian:bullseye-slim
FROM node:18-bullseye-slim as builder-node
RUN apt-get update
RUN apt-get install -y ca-certificates jq netcat
RUN apt-get install -y make
WORKDIR /app
COPY . .
RUN find . -name 'node_modules' | xargs -n1 rm -rf
RUN find . -name 'dist' | xargs -n1 rm -rf
RUN yarn
WORKDIR /app/packages/nitro-gui
RUN VITE_RPC_HOST=CERC_RUNTIME_ENV_RPC_HOST yarn build
RUN if [ -d /app/packages/nitro-auth-gui ]; then \
cd /app/packages/nitro-auth-gui && \
VITE_RPC_URL=CERC_RUNTIME_ENV_RPC_URL VITE_TARGET_URL=CERC_RUNTIME_ENV_TARGET_URL yarn build; \
fi
WORKDIR /app/packages/nitro-auth
RUN if [ -d /app/packages/nitro-auth ]; then cd /app/packages/nitro-auth && yarn build; fi
FROM node:18-bullseye-slim
RUN apt-get update
RUN apt-get install -y ca-certificates jq netcat make curl wget
RUN rm -rf /var/lib/apt/lists/*
RUN npm install -g http-server
RUN npm config set @cerc-io:registry https://git.vdb.to/api/packages/cerc-io/npm/
RUN npm install -g @cerc-io/peer
WORKDIR /app
COPY --from=builder /app/nitro .
COPY --from=builder-node /app /app-node
@@ -26,6 +26,8 @@ RUN \
&& su ${USERNAME} -c "umask 0002 && npm install -g eslint" \
# Install semver
&& su ${USERNAME} -c "umask 0002 && npm install -g semver" \
# Install pnpm
&& su ${USERNAME} -c "umask 0002 && npm install -g pnpm" \
&& npm cache clean --force > /dev/null 2>&1
# [Optional] Uncomment this section to install additional OS packages.
@@ -35,6 +37,9 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# [Optional] Uncomment if you want to install more global node modules
# RUN su node -c "npm install -g <your-package-list-here>"
# We do this to get a yq binary from the published container, for the correct architecture we're building here
COPY --from=docker.io/mikefarah/yq:latest /usr/bin/yq /usr/local/bin/yq
# Expose port for http
EXPOSE 80
@@ -11,8 +11,14 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/nextjs-base:local}
docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
rc=$?
if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
if [ $rc -ne 0 ]; then
echo "BUILD FAILED" 1>&2
exit $rc
fi
if [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
cat <<EOF
#################################################################
@@ -10,10 +10,12 @@ TRG_DIR="${3:-.next-r}"
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=npm
else
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
fi
fi
@@ -9,7 +9,9 @@ CERC_MIN_NEXTVER=13.4.2
CERC_NEXT_VERSION="${CERC_NEXT_VERSION:-keep}"
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ]; then
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
@@ -113,7 +115,7 @@ if [ "$CERC_NEXT_VERSION" != "keep" ] && [ "$CUR_NEXT_VERSION" != "$CERC_NEXT_VE
mv package.json.$$ package.json
fi
$CERC_BUILD_TOOL install || exit 1
time $CERC_BUILD_TOOL install || exit 1
CUR_NEXT_VERSION=`jq -r '.version' node_modules/next/package.json`
@@ -136,9 +138,9 @@ to use for the build with:
EOF
cat package.json | jq ".dependencies.next = \"^$CERC_MIN_NEXTVER\"" > package.json.$$
mv package.json.$$ package.json
$CERC_BUILD_TOOL install || exit 1
time $CERC_BUILD_TOOL install || exit 1
fi
$CERC_BUILD_TOOL run cerc_compile || exit 1
time $CERC_BUILD_TOOL run cerc_compile || exit 1
exit 0
@@ -16,7 +16,9 @@ trap ctrl_c INT
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ] && [ ! -f "package-lock.json" ]; then
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
@@ -4,9 +4,10 @@ RUN apk --update --no-cache add python3 alpine-sdk bash curl jq
WORKDIR /app
COPY . .
COPY packages/nitro-rpc-client .
RUN echo "Installing dependencies" && \
yarn
RUN cd packages/nitro-rpc-client
RUN yarn build
RUN ln -s /app/bin/nitro-rpc-client.js /bin/nitro-rpc-client
@@ -0,0 +1,12 @@
FROM node:18.17.1-alpine3.18
RUN apk --update --no-cache add python3 alpine-sdk bash curl jq
WORKDIR /app
COPY . .
RUN echo "Installing dependencies" && \
yarn && yarn build:node && yarn build:browser
WORKDIR /app
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Build cerc/ts-nitro
source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/ts-nitro:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/ts-nitro
@@ -1,6 +1,6 @@
# Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
# [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
ARG VARIANT=20-bullseye
ARG VARIANT=20-bullseye-slim
FROM node:${VARIANT}
ARG USERNAME=node
@@ -24,6 +24,10 @@ RUN \
&& su ${USERNAME} -c "npm config -g set prefix ${NPM_GLOBAL}" \
# Install eslint
&& su ${USERNAME} -c "umask 0002 && npm install -g eslint" \
# Install semver
&& su ${USERNAME} -c "umask 0002 && npm install -g semver" \
# Install pnpm
&& su ${USERNAME} -c "umask 0002 && npm install -g pnpm" \
&& npm cache clean --force > /dev/null 2>&1
# [Optional] Uncomment this section to install additional OS packages.
@@ -1,11 +1,12 @@
FROM cerc/webapp-base:local as builder
ARG CERC_BUILD_TOOL
ARG CERC_BUILD_OUTPUT_DIR
WORKDIR /app
COPY . .
RUN rm -rf node_modules build .next*
RUN /scripts/build-app.sh /app build /data
RUN rm -rf node_modules build dist .next*
RUN /scripts/build-app.sh /app /data
FROM cerc/webapp-base:local
COPY --from=builder /data /data
@@ -11,8 +11,14 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/webapp-base:local}
docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
rc=$?
if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/webapp-base:local" ]; then
if [ $rc -ne 0 ]; then
echo "BUILD FAILED" 1>&2
exit $rc
fi
if [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/webapp-base:local" ]; then
cat <<EOF
#################################################################
@@ -7,9 +7,10 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
fi
CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
CERC_BUILD_OUTPUT_DIR="${CERC_BUILD_OUTPUT_DIR}"
WORK_DIR="${1:-/app}"
OUTPUT_DIR="${2:-build}"
DEST_DIR="${3:-/data}"
DEST_DIR="${2:-/data}"
if [ -f "${WORK_DIR}/build-webapp.sh" ]; then
echo "Building webapp with ${WORK_DIR}/build-webapp.sh ..."
@@ -22,18 +23,30 @@ elif [ -f "${WORK_DIR}/package.json" ]; then
cd "${WORK_DIR}" || exit 1
if [ -z "$CERC_BUILD_TOOL" ]; then
if [ -f "yarn.lock" ]; then
if [ -f "pnpm-lock.yaml" ]; then
CERC_BUILD_TOOL=pnpm
elif [ -f "yarn.lock" ]; then
CERC_BUILD_TOOL=yarn
else
CERC_BUILD_TOOL=npm
fi
fi
$CERC_BUILD_TOOL install || exit 1
$CERC_BUILD_TOOL build || exit 1
time $CERC_BUILD_TOOL install || exit 1
time $CERC_BUILD_TOOL build || exit 1
rm -rf "${DEST_DIR}"
mv "${WORK_DIR}/${OUTPUT_DIR}" "${DEST_DIR}"
if [ -z "${CERC_BUILD_OUTPUT_DIR}" ]; then
if [ -d "${WORK_DIR}/dist" ]; then
CERC_BUILD_OUTPUT_DIR="${WORK_DIR}/dist"
elif [ -d "${WORK_DIR}/build" ]; then
CERC_BUILD_OUTPUT_DIR="${WORK_DIR}/build"
else
echo "ERROR: Unable to locate build output. Set with --extra-build-args \"--build-arg CERC_BUILD_OUTPUT_DIR=path\"" 1>&2
exit 1
fi
fi
mv "${CERC_BUILD_OUTPUT_DIR}" "${DEST_DIR}"
else
echo "Copying static app ..."
mv "${WORK_DIR}" "${DEST_DIR}"
@@ -3,13 +3,28 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x
fi
CERC_LISTEN_PORT=${CERC_LISTEN_PORT:-80}
CERC_WEBAPP_FILES_DIR="${CERC_WEBAPP_FILES_DIR:-/data}"
CERC_ENABLE_CORS="${CERC_ENABLE_CORS:-false}"
CERC_SINGLE_PAGE_APP="${CERC_SINGLE_PAGE_APP}"
if [ -z "${CERC_SINGLE_PAGE_APP}" ]; then
if [ 1 -eq $(find "${CERC_WEBAPP_FILES_DIR}" -name '*.html' | wc -l) ] && [ -d "${CERC_WEBAPP_FILES_DIR}/static" ]; then
CERC_SINGLE_PAGE_APP=true
else
CERC_SINGLE_PAGE_APP=false
fi
fi
if [ "true" == "$CERC_ENABLE_CORS" ]; then
CERC_HTTP_EXTRA_ARGS="$CERC_HTTP_EXTRA_ARGS --cors"
fi
if [ "true" == "$CERC_SINGLE_PAGE_APP" ]; then
# Create a catchall redirect back to /
CERC_HTTP_EXTRA_ARGS="$CERC_HTTP_EXTRA_ARGS --proxy http://localhost:${CERC_LISTEN_PORT}?"
fi
LACONIC_HOSTED_CONFIG_FILE=${LACONIC_HOSTED_CONFIG_FILE}
if [ -z "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
if [ -f "/config/laconic-hosted-config.yml" ]; then
@@ -20,8 +35,8 @@ if [ -z "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
fi
if [ -f "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
/scripts/apply-webapp-config.sh $LACONIC_HOSTED_CONFIG_FILE ${CERC_WEBAPP_FILES_DIR}
/scripts/apply-webapp-config.sh $LACONIC_HOSTED_CONFIG_FILE "${CERC_WEBAPP_FILES_DIR}"
fi
/scripts/apply-runtime-env.sh ${CERC_WEBAPP_FILES_DIR}
http-server $CERC_HTTP_EXTRA_ARGS -p ${CERC_LISTEN_PORT:-80} ${CERC_WEBAPP_FILES_DIR}
http-server $CERC_HTTP_EXTRA_ARGS -p ${CERC_LISTEN_PORT} "${CERC_WEBAPP_FILES_DIR}"
@@ -0,0 +1,104 @@
# go-nitro-auth
Deploy a stack for demoing Nitro-based auth, using either a local fixturenet (fully self-contained) or remote testnet.
## Local Fixturenet (Self-Contained)
### Clone required repositories
```
$ laconic-so --stack fixturenet-eth setup-repositories
$ laconic-so --stack go-nitro-auth setup-repositories
```
### Build containers
```
$ laconic-so --stack fixturenet-eth build-containers
$ laconic-so --stack go-nitro-auth build-containers
```
### Create a deployment
```
$ laconic-so --stack fixturenet-eth deploy init --output nitro-net.yml
$ laconic-so --stack fixturenet-eth deploy create --spec-file nitro-net.yml --deployment-dir /srv/nitro-net
$ laconic-so --stack go-nitro-auth deploy init --map-ports-to-host any-same --output nitro-auth.yml
$ laconic-so --stack go-nitro-auth deploy create --spec-file nitro-auth.yml --deployment-dir /srv/nitro-auth
# Place them both in the same namespace (TODO: support setting the deployment name via --cluster).
$ cp /srv/nitro-net/deployment.yml /srv/nitro-auth/deployment.yml
```
### Start the containers
```
$ laconic-so deployment --dir /srv/nitro-net up
$ laconic-so deployment --dir /srv/nitro-auth up
```
### Open the webapp
Visit http://localhost:5678
## Remote Testnet
This example will use the Linea Sepolia testnet.
### Clone required repositories
```
$ laconic-so --stack go-nitro-auth setup-repositories
```
### Build containers
```
$ laconic-so --stack go-nitro-auth build-containers
```
### Create a deployment
```
$ laconic-so --stack go-nitro-auth deploy init --map-ports-to-host any-same --output nitro-auth.yml
$ laconic-so --stack go-nitro-auth deploy create --spec-file nitro-auth.yml --deployment-dir /srv/nitro-auth
```
### Set your keys, contract addresses, etc.
You must set the private keys for two accounts with funds on the target network, as well as the contract addresses
(if they already exist) or else an account to create them. You must also set the URL to use for WebSocket connections,
eg, `wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY>`
#### Config
```
$ vim /srv/nitro-auth/config.env
# Addresses of existing contracts.
CERC_CA_ADDRESS="0x1Ae815c3e7556e16ceaB6B6d46306C1870EB6d24"
CERC_NA_ADDRESS="0xc453C5E3f304bb545A3Df7bBa02fe6274A056636"
CERC_VPA_ADDRESS="0xA11af80D75b1150631FA78178c94fa451c7172a8"
# Else the private key of an account and RPC URL to use create them.
CERC_PRIVATE_KEY_DEPLOYER=<PRIV_KEY_HERE>
CERC_ETH_RPC_ENDPOINT=https://rpc.sepolia.linea.build
# The WebSocket chain URL.
CERC_NITRO_CHAIN_URL=wss://linea-sepolia.infura.io/ws/v3/<MY_API_KEY_HERE>
# Private key for "Alice" account (payer)
CERC_NITRO_CHAIN_PK_ALICE=<ALICE_PRIVATE_KEY_HERE>
# Private key for "Bob" account (payee)
CERC_NITRO_CHAIN_PK_BOB=<BOB_PRIVATE_KEY_HERE>
```
### Start the stack
```
$ laconic-so deployment --dir /srv/nitro-auth up
```
### Open the webapp
Visit http://localhost:5678
@@ -0,0 +1,17 @@
version: "1.0"
name: nitro-auth
description: "Stack to demonstrate payments between various services"
repos:
# for nitro-contracts and ts-nitro example app
- github.com/cerc-io/ts-nitro@telackey/demo-auth
# for go-nitro, nitro-rpc-client, and go-nitro-auth example app
- github.com/cerc-io/go-nitro@telackey/update
containers:
# nitro
- cerc/nitro-contracts
- cerc/ts-nitro
- cerc/go-nitro
- cerc/nitro-rpc-client
pods:
- nitro-contracts
- nitro-auth
@@ -33,23 +33,29 @@ laconic-so --stack uniswap-urbit-app deploy init --output uniswap-urbit-app-spec
### Ports
Edit `network` in spec file to map container ports to same ports in host:
Edit `uniswap-urbit-app-spec.yml` such that it looks like:
```yml
...
stack: uniswap-urbit-app
deploy-to: compose
network:
ports:
urbit-fake-ship:
- '8080:80'
proxy-server:
- '4000:4000'
urbit-fake-ship:
- '8080:80'
ipfs:
- '8081:8080'
- '5001:5001'
...
- '4001'
- '8081:8080'
- 0.0.0.0:5001:5001
volumes:
urbit_app_builds: ./data/urbit_app_builds
urbit_data: ./data/urbit_data
ipfs-import: ./data/ipfs-import
ipfs-data: ./data/ipfs-data
```
Note: Skip the `ipfs` ports if need to use an externally running IPFS node
Note: Skip the `ipfs` ports if using an externally running IPFS node, set via `config.env`, below.
### Data volumes
@@ -327,12 +327,14 @@ def init_operation(deploy_command_context, stack, deployer_type, config,
default_spec_file_content = call_stack_deploy_init(deploy_command_context)
spec_file_content = {"stack": stack, constants.deploy_to_key: deployer_type}
if deployer_type == "k8s":
if kube_config is None:
if kube_config:
spec_file_content.update({constants.kube_config_key: kube_config})
else:
error_exit("--kube-config must be supplied with --deploy-to k8s")
if image_registry is None:
error_exit("--image-registry must be supplied with --deploy-to k8s")
spec_file_content.update({constants.kube_config_key: kube_config})
spec_file_content.update({constants.image_registry_key: image_registry})
if image_registry:
spec_file_content.update({constants.image_registry_key: image_registry})
else:
print("WARNING: --image-registry not specified, only default container registries (eg, Docker Hub) will be available")
else:
# Check for --kube-config supplied for non-relevant deployer types
if kube_config is not None:
+23
View File
@@ -29,6 +29,29 @@ def _image_needs_pushed(image: str):
return image.endswith(":local")
def remote_image_exists(remote_repo_url: str, local_tag: str):
docker = DockerClient()
try:
remote_tag = remote_tag_for_image(local_tag, remote_repo_url)
result = docker.manifest.inspect(remote_tag)
return True if result else False
except Exception: # noqa: E722
return False
def add_tags_to_image(remote_repo_url: str, local_tag: str, *additional_tags):
if not additional_tags:
return
if not remote_image_exists(remote_repo_url, local_tag):
raise Exception(f"{local_tag} does not exist in {remote_repo_url}")
docker = DockerClient()
remote_tag = remote_tag_for_image(local_tag, remote_repo_url)
new_remote_tags = [remote_tag_for_image(tag, remote_repo_url) for tag in additional_tags]
docker.buildx.imagetools.create(sources=[remote_tag], tags=new_remote_tags)
def remote_tag_for_image(image: str, remote_repo_url: str):
# Turns image tags of the form: foo/bar:local into remote.repo/org/bar:deploy
major_parts = image.split("/", 2)
+11 -7
View File
@@ -101,7 +101,7 @@ class ClusterInfo:
)
return service
def get_ingress(self, use_tls=False):
def get_ingress(self, use_tls=False, certificate=None, cluster_issuer="letsencrypt-prod"):
# No ingress for a deployment that has no http-proxy defined, for now
http_proxy_info_list = self.spec.get_http_proxy()
ingress = None
@@ -114,8 +114,8 @@ class ClusterInfo:
host_name = http_proxy_info["host-name"]
rules = []
tls = [client.V1IngressTLS(
hosts=[host_name],
secret_name=f"{self.app_name}-tls"
hosts=certificate["spec"]["dnsNames"] if certificate else [host_name],
secret_name=certificate["spec"]["secretName"] if certificate else f"{self.app_name}-tls"
)] if use_tls else None
paths = []
for route in http_proxy_info["routes"]:
@@ -147,13 +147,17 @@ class ClusterInfo:
tls=tls,
rules=rules
)
ingress_annotations = {
"kubernetes.io/ingress.class": "nginx",
}
if not certificate:
ingress_annotations["cert-manager.io/cluster-issuer"] = cluster_issuer
ingress = client.V1Ingress(
metadata=client.V1ObjectMeta(
name=f"{self.app_name}-ingress",
annotations={
"kubernetes.io/ingress.class": "nginx",
"cert-manager.io/cluster-issuer": "letsencrypt-prod"
}
annotations=ingress_annotations
),
spec=spec
)
+45 -3
View File
@@ -169,6 +169,39 @@ class K8sDeployer(Deployer):
print("Service created:")
print(f"{service_resp}")
def _find_certificate_for_host_name(self, host_name):
all_certificates = self.custom_obj_api.list_namespaced_custom_object(
group="cert-manager.io",
version="v1",
namespace=self.k8s_namespace,
plural="certificates"
)
host_parts = host_name.split(".", 1)
host_as_wild = None
if len(host_parts) == 2:
host_as_wild = f"*.{host_parts[1]}"
now = datetime.utcnow().replace(tzinfo=timezone.utc)
fmt = "%Y-%m-%dT%H:%M:%S%z"
# Walk over all the configured certificates.
for cert in all_certificates["items"]:
dns = cert["spec"]["dnsNames"]
# Check for an exact hostname match or a wildcard match.
if host_name in dns or host_as_wild in dns:
status = cert.get("status", {})
# Check the certificate date.
if "notAfter" in status and "notBefore" in status:
before = datetime.strptime(status["notBefore"], fmt)
after = datetime.strptime(status["notAfter"], fmt)
if before < now < after:
# Check the status is Ready
for condition in status.get("conditions", []):
if "True" == condition.get("status") and "Ready" == condition.get("type"):
return cert
return None
def up(self, detach, services):
if not opts.o.dry_run:
if self.is_kind():
@@ -189,8 +222,15 @@ class K8sDeployer(Deployer):
self._create_volume_data()
self._create_deployment()
http_proxy_info = self.cluster_info.spec.get_http_proxy()
# Note: at present we don't support tls for kind (and enabling tls causes errors)
ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=not self.is_kind())
use_tls = http_proxy_info and not self.is_kind()
certificate = self._find_certificate_for_host_name(http_proxy_info[0]["host-name"]) if use_tls else None
if opts.o.debug:
if certificate:
print(f"Using existing certificate: {certificate}")
ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=use_tls, certificate=certificate)
if ingress:
if opts.o.debug:
print(f"Sending this ingress: {ingress}")
@@ -350,9 +390,11 @@ class K8sDeployer(Deployer):
name=ingress.spec.tls[0].secret_name
)
hostname = ingress.spec.tls[0].hosts[0]
hostname = ingress.spec.rules[0].host
ip = ingress.status.load_balancer.ingress[0].ip
tls = "notBefore: %s, notAfter: %s" % (cert["status"]["notBefore"], cert["status"]["notAfter"])
tls = "notBefore: %s; notAfter: %s; names: %s" % (
cert["status"]["notBefore"], cert["status"]["notAfter"], ingress.spec.tls[0].hosts
)
except: # noqa: E722
pass
+22 -24
View File
@@ -27,7 +27,9 @@ class ResourceLimits:
memory: int = None
storage: int = None
def __init__(self, obj={}):
def __init__(self, obj=None):
if obj is None:
obj = {}
if "cpus" in obj:
self.cpus = float(obj["cpus"])
if "memory" in obj:
@@ -50,7 +52,9 @@ class Resources:
limits: ResourceLimits = None
reservations: ResourceLimits = None
def __init__(self, obj={}):
def __init__(self, obj=None):
if obj is None:
obj = {}
if "reservations" in obj:
self.reservations = ResourceLimits(obj["reservations"])
if "limits" in obj:
@@ -72,7 +76,9 @@ class Spec:
obj: typing.Any
file_path: Path
def __init__(self, file_path: Path = None, obj={}) -> None:
def __init__(self, file_path: Path = None, obj=None) -> None:
if obj is None:
obj = {}
self.file_path = file_path
self.obj = obj
@@ -91,49 +97,41 @@ class Spec:
self.file_path = file_path
def get_image_registry(self):
return (self.obj[constants.image_registry_key]
if self.obj and constants.image_registry_key in self.obj
else None)
return self.obj.get(constants.image_registry_key)
def get_volumes(self):
return (self.obj["volumes"]
if self.obj and "volumes" in self.obj
else {})
return self.obj.get(constants.volumes_key, {})
def get_configmaps(self):
return (self.obj["configmaps"]
if self.obj and "configmaps" in self.obj
else {})
return self.obj.get(constants.configmaps_key, {})
def get_container_resources(self):
return Resources(self.obj.get("resources", {}).get("containers", {}))
return Resources(self.obj.get(constants.resources_key, {}).get("containers", {}))
def get_volume_resources(self):
return Resources(self.obj.get("resources", {}).get("volumes", {}))
return Resources(self.obj.get(constants.resources_key, {}).get(constants.volumes_key, {}))
def get_http_proxy(self):
return (self.obj[constants.network_key][constants.http_proxy_key]
if self.obj and constants.network_key in self.obj
and constants.http_proxy_key in self.obj[constants.network_key]
else None)
return self.obj.get(constants.network_key, {}).get(constants.http_proxy_key, [])
def get_annotations(self):
return self.obj.get("annotations", {})
return self.obj.get(constants.annotations_key, {})
def get_labels(self):
return self.obj.get("labels", {})
return self.obj.get(constants.labels_key, {})
def get_privileged(self):
return "true" == str(self.obj.get("security", {}).get("privileged", "false")).lower()
return "true" == str(self.obj.get(constants.security_key, {}).get("privileged", "false")).lower()
def get_capabilities(self):
return self.obj.get("security", {}).get("capabilities", [])
return self.obj.get(constants.security_key, {}).get("capabilities", [])
def get_deployment_type(self):
return self.obj[constants.deploy_to_key]
return self.obj.get(constants.deploy_to_key)
def is_kubernetes_deployment(self):
return self.get_deployment_type() in [constants.k8s_kind_deploy_type, constants.k8s_deploy_type]
return self.get_deployment_type() in [constants.k8s_kind_deploy_type,
constants.k8s_deploy_type]
def is_kind_deployment(self):
return self.get_deployment_type() in [constants.k8s_kind_deploy_type]
@@ -24,8 +24,9 @@ import uuid
import click
from stack_orchestrator.deploy.images import remote_image_exists, add_tags_to_image
from stack_orchestrator.deploy.webapp import deploy_webapp
from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient,
from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient, TimedLogger,
build_container_image, push_container_image,
file_hash, deploy_to_k8s, publish_deployment,
hostname_for_deployment_request, generate_hostname_for_app,
@@ -33,7 +34,6 @@ from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient,
def process_app_deployment_request(
run_id,
ctx,
laconic: LaconicRegistryClient,
app_deployment_request,
@@ -43,13 +43,18 @@ def process_app_deployment_request(
deployment_parent_dir,
kube_config,
image_registry,
log_file=None
force_rebuild,
logger
):
logger.log("BEGIN - process_app_deployment_request")
# 1. look up application
app = laconic.get_record(app_deployment_request.attributes.application, require=True)
logger.log(f"Retrieved app record {app_deployment_request.attributes.application}")
# 2. determine dns
requested_name = hostname_for_deployment_request(app_deployment_request, laconic)
logger.log(f"Determined requested name: {requested_name}")
# HACK
if "." in requested_name:
@@ -67,7 +72,7 @@ def process_app_deployment_request(
matched_owner = match_owner(app_deployment_request, laconic.get_record(dns_record.attributes.request, require=True))
if matched_owner:
print("Matched DnsRecord ownership:", matched_owner)
logger.log(f"Matched DnsRecord ownership: {matched_owner}")
else:
raise Exception("Unable to confirm ownership of DnsRecord %s for request %s" %
(dns_record.id, app_deployment_request.id))
@@ -91,7 +96,9 @@ def process_app_deployment_request(
deployment_record = laconic.get_record(app_deployment_crn)
deployment_dir = os.path.join(deployment_parent_dir, fqdn)
deployment_config_file = os.path.join(deployment_dir, "config.env")
# TODO: Is there any reason not to simplify the hash input to the app_deployment_crn?
deployment_container_tag = "laconic-webapp/%s:local" % hashlib.md5(deployment_dir.encode()).hexdigest()
app_image_shared_tag = f"laconic-webapp/{app.id}:local"
# b. check for deployment directory (create if necessary)
if not os.path.exists(deployment_dir):
if deployment_record:
@@ -106,11 +113,26 @@ def process_app_deployment_request(
needs_k8s_deploy = False
# 6. build container (if needed)
if not deployment_record or deployment_record.attributes.application != app.id:
# TODO: pull from request
extra_build_args = []
build_container_image(app, deployment_container_tag, extra_build_args, log_file)
push_container_image(deployment_dir, log_file)
needs_k8s_deploy = True
# check if the image already exists
shared_tag_exists = remote_image_exists(image_registry, app_image_shared_tag)
if shared_tag_exists and not force_rebuild:
# simply add our unique tag to the existing image and we are done
logger.log(f"Using existing app image {app_image_shared_tag} for {deployment_container_tag}")
add_tags_to_image(image_registry, app_image_shared_tag, deployment_container_tag)
logger.log("Tag complete")
else:
extra_build_args = [] # TODO: pull from request
logger.log(f"Building container image {deployment_container_tag}")
build_container_image(app, deployment_container_tag, extra_build_args, logger)
logger.log("Build complete")
logger.log(f"Pushing container image {deployment_container_tag}")
push_container_image(deployment_dir, logger)
logger.log("Push complete")
# The build/push commands above will use the unique deployment tag, so now we need to add the shared tag.
logger.log(f"Updating app image tag {app_image_shared_tag} from build of {deployment_container_tag}")
add_tags_to_image(image_registry, deployment_container_tag, app_image_shared_tag)
logger.log("Tag complete")
# 7. update config (if needed)
if not deployment_record or file_hash(deployment_config_file) != deployment_record.attributes.meta.config:
@@ -118,13 +140,13 @@ def process_app_deployment_request(
# 8. update k8s deployment
if needs_k8s_deploy:
print("Deploying to k8s")
deploy_to_k8s(
deployment_record,
deployment_dir,
log_file
logger
)
logger.log("Publishing deployment to registry.")
publish_deployment(
laconic,
app,
@@ -133,8 +155,11 @@ def process_app_deployment_request(
dns_record,
dns_crn,
deployment_dir,
app_deployment_request
app_deployment_request,
logger
)
logger.log("Publication complete.")
logger.log("END - process_app_deployment_request")
def load_known_requests(filename):
@@ -171,12 +196,13 @@ def dump_known_requests(filename, requests, status="SEEN"):
@click.option("--dry-run", help="Don't do anything, just report what would be done.", is_flag=True)
@click.option("--include-tags", help="Only include requests with matching tags (comma-separated).", default="")
@click.option("--exclude-tags", help="Exclude requests with matching tags (comma-separated).", default="")
@click.option("--force-rebuild", help="Rebuild even if the image already exists.", is_flag=True)
@click.option("--log-dir", help="Output build/deployment logs to directory.", default=None)
@click.pass_context
def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_dir, # noqa: C901
request_id, discover, state_file, only_update_state,
dns_suffix, record_namespace_dns, record_namespace_deployments, dry_run,
include_tags, exclude_tags, log_dir):
include_tags, exclude_tags, force_rebuild, log_dir):
if request_id and discover:
print("Cannot specify both --request-id and --discover", file=sys.stderr)
sys.exit(2)
@@ -295,8 +321,9 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
run_log_file = open(run_log_file_path, "wt")
run_reg_client = LaconicRegistryClient(laconic_config, log_file=run_log_file)
logger = TimedLogger(run_id, run_log_file)
logger.log("Processing ...")
process_app_deployment_request(
run_id,
ctx,
run_reg_client,
r,
@@ -306,12 +333,15 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
os.path.abspath(deployment_parent_dir),
kube_config,
image_registry,
run_log_file
force_rebuild,
logger
)
status = "DEPLOYED"
except Exception as e:
print("ERROR: " + str(e), file=run_log_file)
logger.log("ERROR: " + str(e))
finally:
if logger:
logger.log(f"DONE with status {status}", show_step_time=False, show_total_time=True)
dump_known_requests(state_file, [r], status)
if run_log_file:
run_log_file.close()
+47 -17
View File
@@ -13,6 +13,7 @@
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import datetime
import hashlib
import json
import os
@@ -39,6 +40,25 @@ class AttrDict(dict):
return v
class TimedLogger:
def __init__(self, id="", file=None):
self.start = datetime.datetime.now()
self.last = self.start
self.id = id
self.file = file
def log(self, msg, show_step_time=True, show_total_time=False):
prefix = f"{datetime.datetime.utcnow()} - {self.id}"
if show_step_time:
prefix += f" - {datetime.datetime.now() - self.last} (step)"
if show_total_time:
prefix += f" - {datetime.datetime.now() - self.start} (total)"
print(f"{prefix}: {msg}", file=self.file)
if self.file:
self.file.flush()
self.last = datetime.datetime.now()
def logged_cmd(log_file, *vargs):
result = None
try:
@@ -219,7 +239,7 @@ def determine_base_container(clone_dir, app_type="webapp"):
return base_container
def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
def build_container_image(app_record, tag, extra_build_args=[], logger=None):
tmpdir = tempfile.mkdtemp()
try:
@@ -228,31 +248,33 @@ def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
repo = random.choice(app_record.attributes.repository)
clone_dir = os.path.join(tmpdir, record_id)
print(f"Cloning repository {repo} to {clone_dir} ...")
logger.log(f"Cloning repository {repo} to {clone_dir} ...")
if ref:
# TODO: Determing branch or hash, and use depth 1 if we can.
git_env = dict(os.environ.copy())
# Never prompt
git_env["GIT_TERMINAL_PROMPT"] = "0"
try:
subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env, stdout=log_file, stderr=log_file)
subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env, stdout=logger.file, stderr=logger.file)
except Exception as e:
print(f"git clone failed. Is the repository {repo} private?", file=log_file)
logger.log(f"git clone failed. Is the repository {repo} private?")
raise e
try:
subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env, stdout=log_file, stderr=log_file)
subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env, stdout=logger.file, stderr=logger.file)
except Exception as e:
print(f"git checkout failed. Does ref {ref} exist?", file=log_file)
logger.log(f"git checkout failed. Does ref {ref} exist?")
raise e
else:
result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir], stdout=log_file, stderr=log_file)
result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir], stdout=logger.file, stderr=logger.file)
result.check_returncode()
base_container = determine_base_container(clone_dir, app_record.attributes.app_type)
print("Building webapp ...")
logger.log("Building webapp ...")
build_command = [
sys.argv[0], "build-webapp",
sys.argv[0],
"--verbose",
"build-webapp",
"--source-repo", clone_dir,
"--tag", tag,
"--base-container", base_container
@@ -261,28 +283,31 @@ def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
build_command.append("--extra-build-args")
build_command.append(" ".join(extra_build_args))
result = subprocess.run(build_command, stdout=log_file, stderr=log_file)
result = subprocess.run(build_command, stdout=logger.file, stderr=logger.file)
result.check_returncode()
finally:
logged_cmd(log_file, "rm", "-rf", tmpdir)
logged_cmd(logger.file, "rm", "-rf", tmpdir)
def push_container_image(deployment_dir, log_file=None):
print("Pushing image ...")
def push_container_image(deployment_dir, logger):
logger.log("Pushing images ...")
result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, "push-images"],
stdout=log_file, stderr=log_file)
stdout=logger.file, stderr=logger.file)
result.check_returncode()
logger.log("Finished pushing images.")
def deploy_to_k8s(deploy_record, deployment_dir, log_file=None):
def deploy_to_k8s(deploy_record, deployment_dir, logger):
if not deploy_record:
command = "up"
else:
command = "update"
logger.log("Deploying to k8s ...")
result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, command],
stdout=log_file, stderr=log_file)
stdout=logger.file, stderr=logger.file)
result.check_returncode()
logger.log("Finished deploying to k8s.")
def publish_deployment(laconic: LaconicRegistryClient,
@@ -292,7 +317,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
dns_record,
dns_crn,
deployment_dir,
app_deployment_request=None):
app_deployment_request=None,
logger=None):
if not deploy_record:
deploy_ver = "0.0.1"
else:
@@ -322,6 +348,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
if app_deployment_request:
new_dns_record["record"]["request"] = app_deployment_request.id
if logger:
logger.log("Publishing DnsRecord.")
dns_id = laconic.publish(new_dns_record, [dns_crn])
new_deployment_record = {
@@ -341,6 +369,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
if app_deployment_request:
new_deployment_record["record"]["request"] = app_deployment_request.id
if logger:
logger.log("Publishing ApplicationDeploymentRecord.")
deployment_id = laconic.publish(new_deployment_record, [deployment_crn])
return {"dns": dns_id, "deployment": deployment_id}
+2 -1
View File
@@ -17,7 +17,7 @@ import click
from stack_orchestrator.command_types import CommandOptions
from stack_orchestrator.repos import setup_repositories
from stack_orchestrator.build import build_containers
from stack_orchestrator.build import build_containers, fetch_containers
from stack_orchestrator.build import build_npms
from stack_orchestrator.build import build_webapp
from stack_orchestrator.deploy.webapp import (run_webapp,
@@ -52,6 +52,7 @@ def cli(ctx, stack, quiet, verbose, dry_run, local_stack, debug, continue_on_err
cli.add_command(setup_repositories.command, "setup-repositories")
cli.add_command(build_containers.command, "build-containers")
cli.add_command(fetch_containers.command, "fetch-containers")
cli.add_command(build_npms.command, "build-npms")
cli.add_command(build_webapp.command, "build-webapp")
cli.add_command(run_webapp.command, "run-webapp")