README describing the workflow around external tests and their repositories

test/externalTests/README.md

@@ -0,0 +1,66 @@
+## Solidity external tests
+This directory contains scripts for compiling some of the popular open-source projects using the
+current version of the compiler and running their test suites.
+
+Since projects often don't use the latest compiler, we keep a fork of each of these projects
+at https://github.com/solidity-external-tests/. If changes are needed to make a project work with the
+latest version of the compiler, they're maintained as a branch on top if the upstream master branch.
+This is especially important for testing our `breaking` branch because we can not realistically expect
+external projects to be instantly compatible with a compiler version that has not been released yet.
+It also isolates us from upstream changes to some degree - their changes won't affect our test suite
+until we explicitly pull them.
+
+### Recommended workflow
+
+#### Adding a new external project
+1. Create a fork of the upstream repository in https://github.com/solidity-external-tests/. If the
+    project consists of several repositories, fork them all.
+2. Remove all the branches except for main one (`master`, `develop`, `main`, etc). This branch is
+    going to be always kept up to date with the upstream repository and should not contain any extra
+    commits.
+3. Create two new versions of the main branch, representing versions of the compiler currently in
+    `develop` and `breaking`. E.g. if the latest Solidity version is 0.7.5 and the main branch of the
+    external project is called `master`, create `master_070` and `master_080`. This is where we will
+    be adding our own commits. The one corresponding to the newer Solidity version should always be
+    on top on the older one. E.g. if a change is needed to keep the project
+    compilable using Solidity 0.7.x, add it in `master_070` and rebase `master_080` on top of it.
+    If it is only needed for the compiler from its `breaking` branch, add it only in `master_080`.
+    - The fewer commits in these branches, the better. Ideally, any changes needed to make the compiler
+        work should be submitted upstream and the branches should just be tracking the upstream
+        one without any extra commits.
+4. Create a script for compiling/testing the project and put it in `test/externalTests/` in the
+    Solidity repository.
+    - The script should apply workarounds necessary to make the project actually use the compiler
+      binary it receives as a parameter and possibly apply some generic workarounds that should
+      work across different versions of the upstream project. Very specific workarounds that may
+      easily break with every upstream change are better done as commits in the fork.
+5. Add the script to `tests/externalTests.sh`.
+6. Add the test to CircleCI configuration. Make sure to add both a compilation-only run and one that
+    also executes the test suite. If the latter takes a significant amount of time (say, more
+    than 15 minutes) make it run nightly rather than on every PR.
+
+#### Pulling upstream changes into a fork
+1. Pull changes directly into the main branch in the fork. This should be straightforward thanks to
+    it not containing any of our customizations.
+2. If the update is straightforward and looks safe, go straight to point 5. Otherwise you need to
+    test it first.
+3. Create new branches corresponding to the two versioned ones but suffixed with `_new`. E.g.
+    `master_070_new` and `master_080_new`. Then rebase them so that they are on top of the updated
+    main branch. This may require tweaking some of the commits to apply our fixes in new places.
+4. Create PRs on `deveop` and `breaking` in Solidity repo which modify the script to use the new
+    branches instead of `master_070`/`master_080`. Tweak the new branches until external tests
+    in CI pass in the PRs.
+5. Discard the PRs and move the original branches to the place where the `_new` ones are and remove
+    the `_new` branches.
+
+#### Changes needed after a breaking release of the compiler
+When a breaking version of the compiler gets released and becomes the most recent release, the scripts
+and the branches in the forks need to be updated:
+- In each fork create a branch corresponding to the new brekaing version. E.g. if the current
+    branches are `master_070` and `master_080` the new one should be called `master_090`.
+- Leave the oldest (i.e. `master_070`) branch as is. We will not be updating it any more but there is no
+    need to remove it.
+- Update scripts on `develop` to now refer to the branch that used to be breaking (i.e. `master_080`)
+    and on `breaking` to refer to the newly added branch (i.e. `master_090`).
+    - Take care not to overwrite these changes when merging `develop` into breaking. Each branch
+        should always use the branches from the external repos that correspond to it.