solidity/test/externalTests/README.md

Solidity external tests

This directory contains scripts for compiling some of the popular open-source projects using the current version of the compiler and running their test suites.

Since projects often do not use the latest compiler, we keep a fork of each of these projects at https://github.com/solidity-external-tests/. If changes are needed to make a project work with the latest version of the compiler, they are maintained as a branch on top of the upstream master branch. This is especially important for testing our breaking branch because we can not realistically expect external projects to be instantly compatible with a compiler version that has not been released yet. It also isolates us from upstream changes to some degree - their changes will not affect our test suite until we explicitly pull them.

Recommended workflow

Adding a new external project

1. Create a fork of the upstream repository in https://github.com/solidity-external-tests/. If the project consists of several repositories, fork them all.
2. Remove all the branches except for main one (master, develop, main, etc). This branch is going to be always kept up to date with the upstream repository and should not contain any extra commits.
3. Create two new versions of the main branch, representing versions of the compiler currently in develop and breaking. E.g. if the latest Solidity version is 0.7.5 and the main branch of the external project is called master, create master_070 and master_080. This is where we will be adding our own commits. The one corresponding to the newer Solidity version should always be rebased on top of the older one. E.g. if a change is needed to keep the project compilable using Solidity 0.7.x, add it in master_070 and rebase master_080 on top of it. If it is only needed for the compiler from its breaking branch, add it only in master_080.
   • The fewer commits in these branches, the better. Ideally, any changes needed to make the compiler work should be submitted upstream and the branches should just be tracking the upstream one without any extra commits.
4. Create a script for compiling/testing the project and put it in test/externalTests/ in the Solidity repository.
   • The script should apply workarounds necessary to make the project actually use the compiler binary it receives as a parameter and possibly apply some generic workarounds that should work across different versions of the upstream project. Very specific workarounds that may easily break with every upstream change are better done as commits in the fork.
5. Add the script to tests/externalTests.sh.
6. Add the test to CircleCI configuration. Make sure to add both a compilation-only run and one that also executes the test suite. If the latter takes a significant amount of time (say, more than 15 minutes) make it run nightly rather than on every PR.

Pulling upstream changes into a fork

1. Pull changes directly into the main branch in the fork. This should be straightforward thanks to it not containing any of our customizations.
2. If the update is straightforward and looks safe, go straight to point 5. Otherwise you need to test it first.
3. Create new branches corresponding to the two versioned ones but suffixed with _new. E.g. master_070_new and master_080_new. Then rebase them so that they are on top of the updated main branch. This may require tweaking some of the commits to apply our fixes in new places.
4. Create PRs on develop and breaking in Solidity repo which modify the script to use the new branches instead of master_070/master_080. Tweak the new branches until external tests in CI pass in the PRs.
5. Discard the PRs and move the original branches to the place where the _new ones are and remove the _new branches.

Changes needed after a breaking release of the compiler

When a breaking version of the compiler gets released and becomes the most recent release, the scripts and the branches in the forks need to be updated:

• In each fork create a branch corresponding to the new breaking version. E.g. if the current branches are master_070 and master_080 the new one should be called master_090.
• Leave the oldest (i.e. master_070) branch as is. We will not be updating it any more but there is no need to remove it.
• Update scripts on develop to now refer to the branch that used to be breaking (i.e. master_080) and on breaking to refer to the newly added branch (i.e. master_090).
  • Take care not to overwrite these changes when merging develop into breaking. Each branch should always use the branches from the external repos that correspond to it.