|
|
|
@@ -11,28 +11,28 @@ PROPOSED Partially Implemented
|
|
|
|
|
## Abstract
|
|
|
|
|
|
|
|
|
|
Recent work in the SDK aimed at breaking apart the monolithic root go module has highlighted
|
|
|
|
|
shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common
|
|
|
|
|
shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common
|
|
|
|
|
language for talking about test scopes and proposes an ideal state of tests at each scope.
|
|
|
|
|
|
|
|
|
|
## Context
|
|
|
|
|
|
|
|
|
|
[ADR-053: Go Module Refactoring](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-053-go-module-refactoring.md) expresses our desire for an SDK composed of many
|
|
|
|
|
independently versioned Go modules, and [ADR-057: App Wiring Part I](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-057-app-wiring-1.md) offers a methodology
|
|
|
|
|
for breaking apart inter-module dependencies through the use of dependency injection. As
|
|
|
|
|
for breaking apart inter-module dependencies through the use of dependency injection. As
|
|
|
|
|
described in [EPIC: Separate all SDK modules into standalone go modules](https://github.com/cosmos/cosmos-sdk/issues/11899), module
|
|
|
|
|
dependencies are particularly complected in the test phase, where simapp is used as
|
|
|
|
|
the key test fixture in setting up and running tests. It is clear that the successful
|
|
|
|
|
the key test fixture in setting up and running tests. It is clear that the successful
|
|
|
|
|
completion of Phases 3 and 4 in that EPIC require the resolution of this dependency problem.
|
|
|
|
|
|
|
|
|
|
In [EPIC: Unit Testing of Modules via Mocks](https://github.com/cosmos/cosmos-sdk/issues/12398) it was thought this Gordian knot could be
|
|
|
|
|
unwound by mocking all dependencies in the test phase for each module, but seeing how these
|
|
|
|
|
refactors were complete rewrites of test suites discussions began around the fate of the
|
|
|
|
|
existing integration tests. One perspective is that they ought to be thrown out, another is
|
|
|
|
|
existing integration tests. One perspective is that they ought to be thrown out, another is
|
|
|
|
|
that integration tests have some utility of their own and a place in the SDK's testing story.
|
|
|
|
|
|
|
|
|
|
Another point of confusion has been the current state of CLI test suites, [x/auth](https://github.com/cosmos/cosmos-sdk/blob/0f7e56c6f9102cda0ca9aba5b6f091dbca976b5a/x/auth/client/testutil/suite.go#L44-L49) for
|
|
|
|
|
example. In code these are called integration tests, but in reality function as end to end
|
|
|
|
|
tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify
|
|
|
|
|
example. In code these are called integration tests, but in reality function as end to end
|
|
|
|
|
tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify
|
|
|
|
|
CLI tests](https://github.com/cosmos/cosmos-sdk/issues/12696) identifies the ideal state of CLI tests using mocks, but does not address the
|
|
|
|
|
place end to end tests may have in the SDK.
|
|
|
|
|
|
|
|
|
@@ -43,8 +43,8 @@ ideal state in the SDK.
|
|
|
|
|
### Unit tests
|
|
|
|
|
|
|
|
|
|
Unit tests exercise the code contained in a single module (e.g. `/x/bank`) or package
|
|
|
|
|
(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two
|
|
|
|
|
levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on
|
|
|
|
|
(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two
|
|
|
|
|
levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on
|
|
|
|
|
[The BDD Books - Formulation](https://leanpub.com/bddbooks-formulation) section 1.3.
|
|
|
|
|
|
|
|
|
|
*Illustrative* tests exercise an atomic part of a module in isolation - in this case we
|
|
|
|
@@ -65,11 +65,11 @@ supplied to the keeper constructor.
|
|
|
|
|
|
|
|
|
|
#### Limitations
|
|
|
|
|
|
|
|
|
|
Certain modules are tightly coupled beyond the test phase. A recent dependency report for
|
|
|
|
|
Certain modules are tightly coupled beyond the test phase. A recent dependency report for
|
|
|
|
|
`bank -> auth` found 274 total usages of `auth` in `bank`, 50 of which are in
|
|
|
|
|
production code and 224 in test. This tight coupling may suggest that either the modules
|
|
|
|
|
production code and 224 in test. This tight coupling may suggest that either the modules
|
|
|
|
|
should be merged, or refactoring is required to abstract references to the core types tying
|
|
|
|
|
the modules together. It could also indicate that these modules should be tested together
|
|
|
|
|
the modules together. It could also indicate that these modules should be tested together
|
|
|
|
|
in integration tests beyond mocked unit tests.
|
|
|
|
|
|
|
|
|
|
In some cases setting up a test case for a module with many mocked dependencies can be quite
|
|
|
|
@@ -82,18 +82,18 @@ Integration tests define and exercise relationships between an arbitrary number
|
|
|
|
|
and/or application subsystems.
|
|
|
|
|
|
|
|
|
|
Wiring for integration tests is provided by `depinject` and some [helper code](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/testutil/sims/app_helpers.go#L95) starts up
|
|
|
|
|
a running application. A section of the running application may then be tested. Certain
|
|
|
|
|
a running application. A section of the running application may then be tested. Certain
|
|
|
|
|
inputs during different phases of the application life cycle are expected to produce
|
|
|
|
|
invariant outputs without too much concern for component internals. This type of black box
|
|
|
|
|
invariant outputs without too much concern for component internals. This type of black box
|
|
|
|
|
testing has a larger scope than unit testing.
|
|
|
|
|
|
|
|
|
|
Example 1 [client/grpc_query_test/TestGRPCQuery](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/client/grpc_query_test.go#L111-L129) - This test is misplaced in `/client`,
|
|
|
|
|
but tests the life cycle of (at least) `runtime` and `bank` as they progress through
|
|
|
|
|
startup, genesis and query time. It also exercises the fitness of the client and query
|
|
|
|
|
startup, genesis and query time. It also exercises the fitness of the client and query
|
|
|
|
|
server without putting bytes on the wire through the use of [QueryServiceTestHelper](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/baseapp/grpcrouter_helpers.go#L31).
|
|
|
|
|
|
|
|
|
|
Example 2 `x/evidence` Keeper integration tests - Starts up an application composed of [8
|
|
|
|
|
modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite
|
|
|
|
|
modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite
|
|
|
|
|
exercises [HandleEquivocationEvidence](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/infraction_test.go#L42) which contains many interactions with the staking
|
|
|
|
|
keeper.
|
|
|
|
|
|
|
|
|
@@ -103,10 +103,10 @@ YAML as above) [statically](https://github.com/cosmos/cosmos-sdk/blob/main/x/nft
|
|
|
|
|
#### Limitations
|
|
|
|
|
|
|
|
|
|
Setting up a particular input state may be more challenging since the application is
|
|
|
|
|
starting from a zero state. Some of this may be addressed by good test fixture
|
|
|
|
|
abstractions with testing of their own. Tests may also be more brittle, and larger
|
|
|
|
|
starting from a zero state. Some of this may be addressed by good test fixture
|
|
|
|
|
abstractions with testing of their own. Tests may also be more brittle, and larger
|
|
|
|
|
refactors could impact application initialization in unexpected ways with harder to
|
|
|
|
|
understand errors. This could also be seen as a benefit, and indeed the SDK's current
|
|
|
|
|
understand errors. This could also be seen as a benefit, and indeed the SDK's current
|
|
|
|
|
integration tests were helpful in tracking down logic errors during earlier stages
|
|
|
|
|
of app-wiring refactors.
|
|
|
|
|
|
|
|
|
@@ -114,38 +114,38 @@ of app-wiring refactors.
|
|
|
|
|
|
|
|
|
|
Simulations (also called generative testing) are a special case of integration tests where
|
|
|
|
|
deterministically random module operations are executed against a running simapp, building
|
|
|
|
|
blocks on the chain until a specified height is reached. No *specific* assertions are
|
|
|
|
|
blocks on the chain until a specified height is reached. No *specific* assertions are
|
|
|
|
|
made for the state transitions resulting from module operations but any error will halt and
|
|
|
|
|
fail the simulation. Since `crisis` is included in simapp and the simulation runs
|
|
|
|
|
fail the simulation. Since `crisis` is included in simapp and the simulation runs
|
|
|
|
|
EndBlockers at the end of each block any module invariant violations will also fail
|
|
|
|
|
the simulation.
|
|
|
|
|
|
|
|
|
|
Modules must implement [AppModuleSimulation.WeightedOperations](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/types/module/simulation.go#L31) to define their
|
|
|
|
|
simulation operations. Note that not all modules implement this which may indicate a
|
|
|
|
|
simulation operations. Note that not all modules implement this which may indicate a
|
|
|
|
|
gap in current simulation test coverage.
|
|
|
|
|
|
|
|
|
|
Modules not returning simulation operations:
|
|
|
|
|
Modules not returning simulation operations:
|
|
|
|
|
|
|
|
|
|
- `auth`
|
|
|
|
|
- `capability`
|
|
|
|
|
- `evidence`
|
|
|
|
|
- `mint`
|
|
|
|
|
- `params`
|
|
|
|
|
* `auth`
|
|
|
|
|
* `capability`
|
|
|
|
|
* `evidence`
|
|
|
|
|
* `mint`
|
|
|
|
|
* `params`
|
|
|
|
|
|
|
|
|
|
A separate binary, [runsim](https://github.com/cosmos/tools/tree/master/cmd/runsim), is responsible for kicking off some of these tests and
|
|
|
|
|
managing their life cycle.
|
|
|
|
|
|
|
|
|
|
#### Limitations
|
|
|
|
|
|
|
|
|
|
- [A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI.
|
|
|
|
|
- [Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why.
|
|
|
|
|
- Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run
|
|
|
|
|
* [A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI.
|
|
|
|
|
* [Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why.
|
|
|
|
|
* Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run
|
|
|
|
|
the simulation locally to reproduce.
|
|
|
|
|
|
|
|
|
|
### E2E tests
|
|
|
|
|
|
|
|
|
|
End to end tests exercise the entire system as we understand it in as close an approximation
|
|
|
|
|
to a production environment as is practical. Presently these tests are located at
|
|
|
|
|
to a production environment as is practical. Presently these tests are located at
|
|
|
|
|
[tests/e2e](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e) and rely on [testutil/network](https://github.com/cosmos/cosmos-sdk/tree/main/testutil/network) to start up an in-process Tendermint node.
|
|
|
|
|
|
|
|
|
|
#### Limitations
|
|
|
|
@@ -164,13 +164,13 @@ The scope of e2e tests has been complected with command line interface testing.
|
|
|
|
|
We accept these test scopes and identify the following decisions points for each.
|
|
|
|
|
|
|
|
|
|
| Scope | App Fixture | Mocks? |
|
|
|
|
|
|-------------|-------------|--------|
|
|
|
|
|
| ----------- | ----------- | ------ |
|
|
|
|
|
| Unit | None | Yes |
|
|
|
|
|
| Integration | depinject | Some |
|
|
|
|
|
| Simulation | simapp | No |
|
|
|
|
|
| Simulation | depinject | No |
|
|
|
|
|
| E2E | simapp | No |
|
|
|
|
|
|
|
|
|
|
#### Unit Tests
|
|
|
|
|
### Unit Tests
|
|
|
|
|
|
|
|
|
|
All modules must have mocked unit test coverage.
|
|
|
|
|
|
|
|
|
@@ -185,11 +185,11 @@ production code.
|
|
|
|
|
|
|
|
|
|
When module unit test introduction as per [EPIC: Unit testing of modules via mocks](https://github.com/cosmos/cosmos-sdk/issues/12398)
|
|
|
|
|
results in a near complete rewrite of an integration test suite the test suite should be
|
|
|
|
|
retained and moved to `/tests/integration`. We accept the resulting test logic
|
|
|
|
|
retained and moved to `/tests/integration`. We accept the resulting test logic
|
|
|
|
|
duplication but recommend improving the unit test suite through the addition of
|
|
|
|
|
illustrative tests.
|
|
|
|
|
|
|
|
|
|
#### Integration Tests
|
|
|
|
|
### Integration Tests
|
|
|
|
|
|
|
|
|
|
All integration tests shall be located in `/tests/integration`, even those which do not
|
|
|
|
|
introduce extra module dependencies.
|
|
|
|
@@ -199,11 +199,11 @@ modules in application startup, i.e. don't depend on simapp.
|
|
|
|
|
|
|
|
|
|
Integration tests should outnumber e2e tests.
|
|
|
|
|
|
|
|
|
|
#### Simulations
|
|
|
|
|
### Simulations
|
|
|
|
|
|
|
|
|
|
Simulations shall startup and test simapp directly.
|
|
|
|
|
Simulations shall use `depinject`. They are located under `/x/{moduleName}/simulation`.
|
|
|
|
|
|
|
|
|
|
#### E2E Tests
|
|
|
|
|
### E2E Tests
|
|
|
|
|
|
|
|
|
|
Existing e2e tests shall be migrated to integration tests by removing the dependency on the
|
|
|
|
|
test network and in-process Tendermint node to ensure we do not lose test coverage.
|
|
|
|
@@ -220,22 +220,22 @@ demonstrated in [PR#12706](https://github.com/cosmos/cosmos-sdk/pull/12706).
|
|
|
|
|
|
|
|
|
|
### Positive
|
|
|
|
|
|
|
|
|
|
- test coverage is increased
|
|
|
|
|
- test organization is improved
|
|
|
|
|
- reduced dependency graph size in modules
|
|
|
|
|
- simapp removed as a dependency from modules
|
|
|
|
|
- inter-module dependencies introduced in test code are removed
|
|
|
|
|
- reduced CI run time after transitioning away from in process Tendermint
|
|
|
|
|
* test coverage is increased
|
|
|
|
|
* test organization is improved
|
|
|
|
|
* reduced dependency graph size in modules
|
|
|
|
|
* simapp removed as a dependency from modules
|
|
|
|
|
* inter-module dependencies introduced in test code are removed
|
|
|
|
|
* reduced CI run time after transitioning away from in process Tendermint
|
|
|
|
|
|
|
|
|
|
### Negative
|
|
|
|
|
|
|
|
|
|
- some test logic duplication between unit and integration tests during transition
|
|
|
|
|
- test written using dockertest DX may be a bit worse
|
|
|
|
|
* some test logic duplication between unit and integration tests during transition
|
|
|
|
|
* test written using dockertest DX may be a bit worse
|
|
|
|
|
|
|
|
|
|
### Neutral
|
|
|
|
|
|
|
|
|
|
- learning curve for BDD style tests
|
|
|
|
|
- some discovery required for e2e transition to dockertest
|
|
|
|
|
* learning curve for BDD style tests
|
|
|
|
|
* some discovery required for e2e transition to dockertest
|
|
|
|
|
|
|
|
|
|
## Further Discussions
|
|
|
|
|
|
|
|
|
@@ -243,8 +243,8 @@ It may be useful if test suites could be run in integration mode (with mocked te
|
|
|
|
|
with e2e fixtures (with real tendermint and many nodes). Integration fixtures could be used
|
|
|
|
|
for quicker runs, e2e fixures could be used for more battle hardening.
|
|
|
|
|
|
|
|
|
|
A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847)
|
|
|
|
|
is in progress for unit tests demonstrating BDD.
|
|
|
|
|
A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847)
|
|
|
|
|
is in progress for unit tests demonstrating BDD [Rejected].
|
|
|
|
|
Observing that a strength of BDD specifications is their readability, and a con is the
|
|
|
|
|
cognitive load while writing and maintaining, current consensus is to reserve BDD use
|
|
|
|
|
for places in the SDK where complex rules and module interactions are demonstrated.
|
|
|
|
|
Julien Robert
