cerc-io/solidity
10
0
Fork 0
mirror of https://github.com/ethereum/solidity synced 2023-10-03 13:03:40 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #10556 from ethereum/external-test-readme

Browse Source 
README describing the workflow around external tests and their repositories
This commit is contained in:
Harikrishnan Mulackal 2020-12-11 17:37:38 +01:00 committed by GitHub
commit 2139f93410
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 92 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

92
test/externalTests/README.md Normal file
View File

@ -0,0 +1,92 @@
## Solidity external tests
This directory contains scripts for compiling some of the popular open-source projects using the
current version of the compiler and running their test suites.
Since projects often do not use the latest compiler, we keep a fork of each of these projects
at https://github.com/solidity-external-tests/. If changes are needed to make a project work with the
latest version of the compiler, they are maintained as a branch on top of the upstream master branch.
This is especially important for testing our `breaking` branch because we can not realistically expect
external projects to be instantly compatible with a compiler version that has not been released yet.
Applying necessary changes ourselves gives us confidence that breaking changes are sane and that
these projects *can* be upgraded at all.
### Recommended workflow
#### Adding a new external project
1. Create a fork of the upstream repository in https://github.com/solidity-external-tests/. If the
project consists of several repositories, fork them all.
2. Remove all the branches except for main one (`master`, `develop`, `main`, etc). This branch is
going to be always kept up to date with the upstream repository and should not contain any extra
commits.
- If the project is not up to date with the latest compiler version but has a branch that is,
try to use that branch instead.
3. Create a new branch named after the main branch and the compiler version from our `develop`
branch. E.g. if the latest Solidity version is 0.7.5 and the main branch of the external project
is called `master`, create `master_070`. This is where we will be adding our own commits.
4. Create a script for compiling/testing the project and put it in `test/externalTests/` in the
Solidity repository.
- The script should apply workarounds necessary to make the project actually use the compiler
binary it receives as a parameter and possibly add generic workarounds that should
work across different versions of the upstream project.
- Very specific workarounds that may easily break with every upstream change are better done as
commits in the newly added branch in the fork instead.
5. List the script in `test/externalTests.sh`.
6. Add the script to CircleCI configuration. Make sure to add both a compilation-only run and one that
also executes the test suite. If the latter takes a significant amount of time (say, more than
15 minutes) make it run nightly rather than on every PR.
7. Make sure that tests pass both on `develop` and on `breaking`. If the compiler from `breaking`
branch will not work without additional changes, add another branch, called after it in turn,
and add necessary workarounds there. Continuing the example above, the new branch would be
called `master_080` and should be rebased on top of `master_070`.
- The fewer commits in these branches, the better. Ideally, any changes needed to make the compiler
work should be submitted upstream and our branches should just be tracking the main upstream
branch without any extra commits.
#### Updating external projects for a PR that introduces breaking changes in the compiler
If a PR to our `breaking` branch introduces changes that will make an external project no longer
compile or pass its tests, the fork needs to be modified:
- If a branch specific to the compiler version from `breaking` does not exist yet:
1. Create the branch. It should be based on the version-specific branch used on `develop`.
2. Make your PR modify the project script in `test/externalScripts/` to use the new branch.
3. You are free to add any changes you need in the new branch since it will not interfere with
tests on `breaking`.
4. Work on your PR until it is approved and merged into `breaking`.
- If the branch already exists and our CI depends on it:
1. If the external project after your changes can still work with `breaking` even without your PR or
if you know that the PR is straightforward and will be merged immediately without interfering
with tests on `breaking` for a significant amount of time, you can just push your modifications
to the branch directly and skip straight to steps 4. and 6.
2. Create a PR in the fork, targeting the existing version-specific branch.
3. In your PR to `breaking`, modify the corresponding script in `test/externalScripts/` to
use the branch from your PR in the fork.
4. Work on your PR until it is approved and ready to merge.
5. Merge the PR in the fork.
6. Discard your changes to the script and merge your PR into `breaking`.
#### Pulling upstream changes into a fork
1. Pull changes directly into the main branch in the fork. This should be straightforward thanks to
it not containing any of our customizations.
2. If the project has been updated to a newer Solidity version, abandon the current version-specific
branch used on `develop` (but do not delete it) and create a new one corresponding to the newer
version. Then update project script in `test/externalTests/` to use the new branch. E.g. if `develop` uses
`master_050` and the project has been updated to use Solidity 0.7.3, create `master_070`.
3. Otherwise, rebase the current version-specific branch on the main branch of the fork. This may require
tweaking some of the commits to apply our fixes in new places.
4. If we have a separate branch for `breaking`, rebase it on top of the one used on `develop`.
The above is the workflow to use when the update is straightforward and looks safe. In that case it is
fine to just modify the branches directly. If this is not the case, it is recommended to first perform the
operation on copies of these version-specific branches and test them by creating PRs on `develop` and
`breaking` to see if tests pass. The PRs should just modify project scripts in `test/externalScripts/`
to use the updated copies of the branches and can be discarded aferwards without being merged.
#### Changes needed after a breaking release of the compiler
When a non-backwards-compatible version becomes the most recent release, `breaking` branch
gets merged into `develop` which automatically results in a switch to the newer version-specific
branches if they exist. If no changes on our part were necessary, it is completely fine to keep using
e.g. the `master_060` of an external project in in Solidity 0.8.x.
Since each project is handled separately, this approach may result in a mix of version-specific branches
between different external projects. For example, in one project we could could have `master_050` on
both `develop` and `breaking` and in another `breaking` could use `master_080` while `develop` still
uses `master_060`.