diff --git a/docs/architecture/adr-059-test-scopes.md b/docs/architecture/adr-059-test-scopes.md index ce08668809..661a1bbd97 100644 --- a/docs/architecture/adr-059-test-scopes.md +++ b/docs/architecture/adr-059-test-scopes.md @@ -11,28 +11,28 @@ PROPOSED Partially Implemented ## Abstract Recent work in the SDK aimed at breaking apart the monolithic root go module has highlighted -shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common +shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common language for talking about test scopes and proposes an ideal state of tests at each scope. ## Context [ADR-053: Go Module Refactoring](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-053-go-module-refactoring.md) expresses our desire for an SDK composed of many independently versioned Go modules, and [ADR-057: App Wiring Part I](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-057-app-wiring-1.md) offers a methodology -for breaking apart inter-module dependencies through the use of dependency injection. As +for breaking apart inter-module dependencies through the use of dependency injection. As described in [EPIC: Separate all SDK modules into standalone go modules](https://github.com/cosmos/cosmos-sdk/issues/11899), module dependencies are particularly complected in the test phase, where simapp is used as -the key test fixture in setting up and running tests. It is clear that the successful +the key test fixture in setting up and running tests. It is clear that the successful completion of Phases 3 and 4 in that EPIC require the resolution of this dependency problem. In [EPIC: Unit Testing of Modules via Mocks](https://github.com/cosmos/cosmos-sdk/issues/12398) it was thought this Gordian knot could be unwound by mocking all dependencies in the test phase for each module, but seeing how these refactors were complete rewrites of test suites discussions began around the fate of the -existing integration tests. One perspective is that they ought to be thrown out, another is +existing integration tests. One perspective is that they ought to be thrown out, another is that integration tests have some utility of their own and a place in the SDK's testing story. Another point of confusion has been the current state of CLI test suites, [x/auth](https://github.com/cosmos/cosmos-sdk/blob/0f7e56c6f9102cda0ca9aba5b6f091dbca976b5a/x/auth/client/testutil/suite.go#L44-L49) for -example. In code these are called integration tests, but in reality function as end to end -tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify +example. In code these are called integration tests, but in reality function as end to end +tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify CLI tests](https://github.com/cosmos/cosmos-sdk/issues/12696) identifies the ideal state of CLI tests using mocks, but does not address the place end to end tests may have in the SDK. @@ -43,8 +43,8 @@ ideal state in the SDK. ### Unit tests Unit tests exercise the code contained in a single module (e.g. `/x/bank`) or package -(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two -levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on +(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two +levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on [The BDD Books - Formulation](https://leanpub.com/bddbooks-formulation) section 1.3. *Illustrative* tests exercise an atomic part of a module in isolation - in this case we @@ -65,11 +65,11 @@ supplied to the keeper constructor. #### Limitations -Certain modules are tightly coupled beyond the test phase. A recent dependency report for +Certain modules are tightly coupled beyond the test phase. A recent dependency report for `bank -> auth` found 274 total usages of `auth` in `bank`, 50 of which are in -production code and 224 in test. This tight coupling may suggest that either the modules +production code and 224 in test. This tight coupling may suggest that either the modules should be merged, or refactoring is required to abstract references to the core types tying -the modules together. It could also indicate that these modules should be tested together +the modules together. It could also indicate that these modules should be tested together in integration tests beyond mocked unit tests. In some cases setting up a test case for a module with many mocked dependencies can be quite @@ -82,18 +82,18 @@ Integration tests define and exercise relationships between an arbitrary number and/or application subsystems. Wiring for integration tests is provided by `depinject` and some [helper code](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/testutil/sims/app_helpers.go#L95) starts up -a running application. A section of the running application may then be tested. Certain +a running application. A section of the running application may then be tested. Certain inputs during different phases of the application life cycle are expected to produce -invariant outputs without too much concern for component internals. This type of black box +invariant outputs without too much concern for component internals. This type of black box testing has a larger scope than unit testing. Example 1 [client/grpc_query_test/TestGRPCQuery](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/client/grpc_query_test.go#L111-L129) - This test is misplaced in `/client`, but tests the life cycle of (at least) `runtime` and `bank` as they progress through -startup, genesis and query time. It also exercises the fitness of the client and query +startup, genesis and query time. It also exercises the fitness of the client and query server without putting bytes on the wire through the use of [QueryServiceTestHelper](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/baseapp/grpcrouter_helpers.go#L31). Example 2 `x/evidence` Keeper integration tests - Starts up an application composed of [8 -modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite +modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite exercises [HandleEquivocationEvidence](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/infraction_test.go#L42) which contains many interactions with the staking keeper. @@ -103,10 +103,10 @@ YAML as above) [statically](https://github.com/cosmos/cosmos-sdk/blob/main/x/nft #### Limitations Setting up a particular input state may be more challenging since the application is -starting from a zero state. Some of this may be addressed by good test fixture -abstractions with testing of their own. Tests may also be more brittle, and larger +starting from a zero state. Some of this may be addressed by good test fixture +abstractions with testing of their own. Tests may also be more brittle, and larger refactors could impact application initialization in unexpected ways with harder to -understand errors. This could also be seen as a benefit, and indeed the SDK's current +understand errors. This could also be seen as a benefit, and indeed the SDK's current integration tests were helpful in tracking down logic errors during earlier stages of app-wiring refactors. @@ -114,38 +114,38 @@ of app-wiring refactors. Simulations (also called generative testing) are a special case of integration tests where deterministically random module operations are executed against a running simapp, building -blocks on the chain until a specified height is reached. No *specific* assertions are +blocks on the chain until a specified height is reached. No *specific* assertions are made for the state transitions resulting from module operations but any error will halt and -fail the simulation. Since `crisis` is included in simapp and the simulation runs +fail the simulation. Since `crisis` is included in simapp and the simulation runs EndBlockers at the end of each block any module invariant violations will also fail the simulation. Modules must implement [AppModuleSimulation.WeightedOperations](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/types/module/simulation.go#L31) to define their -simulation operations. Note that not all modules implement this which may indicate a +simulation operations. Note that not all modules implement this which may indicate a gap in current simulation test coverage. -Modules not returning simulation operations: +Modules not returning simulation operations: -- `auth` -- `capability` -- `evidence` -- `mint` -- `params` +* `auth` +* `capability` +* `evidence` +* `mint` +* `params` A separate binary, [runsim](https://github.com/cosmos/tools/tree/master/cmd/runsim), is responsible for kicking off some of these tests and managing their life cycle. #### Limitations -- [A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI. -- [Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why. -- Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run +* [A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI. +* [Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why. +* Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run the simulation locally to reproduce. ### E2E tests End to end tests exercise the entire system as we understand it in as close an approximation -to a production environment as is practical. Presently these tests are located at +to a production environment as is practical. Presently these tests are located at [tests/e2e](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e) and rely on [testutil/network](https://github.com/cosmos/cosmos-sdk/tree/main/testutil/network) to start up an in-process Tendermint node. #### Limitations @@ -164,13 +164,13 @@ The scope of e2e tests has been complected with command line interface testing. We accept these test scopes and identify the following decisions points for each. | Scope | App Fixture | Mocks? | -|-------------|-------------|--------| +| ----------- | ----------- | ------ | | Unit | None | Yes | | Integration | depinject | Some | -| Simulation | simapp | No | +| Simulation | depinject | No | | E2E | simapp | No | -#### Unit Tests +### Unit Tests All modules must have mocked unit test coverage. @@ -185,11 +185,11 @@ production code. When module unit test introduction as per [EPIC: Unit testing of modules via mocks](https://github.com/cosmos/cosmos-sdk/issues/12398) results in a near complete rewrite of an integration test suite the test suite should be -retained and moved to `/tests/integration`. We accept the resulting test logic +retained and moved to `/tests/integration`. We accept the resulting test logic duplication but recommend improving the unit test suite through the addition of illustrative tests. -#### Integration Tests +### Integration Tests All integration tests shall be located in `/tests/integration`, even those which do not introduce extra module dependencies. @@ -199,11 +199,11 @@ modules in application startup, i.e. don't depend on simapp. Integration tests should outnumber e2e tests. -#### Simulations +### Simulations -Simulations shall startup and test simapp directly. +Simulations shall use `depinject`. They are located under `/x/{moduleName}/simulation`. -#### E2E Tests +### E2E Tests Existing e2e tests shall be migrated to integration tests by removing the dependency on the test network and in-process Tendermint node to ensure we do not lose test coverage. @@ -220,22 +220,22 @@ demonstrated in [PR#12706](https://github.com/cosmos/cosmos-sdk/pull/12706). ### Positive -- test coverage is increased -- test organization is improved -- reduced dependency graph size in modules -- simapp removed as a dependency from modules -- inter-module dependencies introduced in test code are removed -- reduced CI run time after transitioning away from in process Tendermint +* test coverage is increased +* test organization is improved +* reduced dependency graph size in modules +* simapp removed as a dependency from modules +* inter-module dependencies introduced in test code are removed +* reduced CI run time after transitioning away from in process Tendermint ### Negative -- some test logic duplication between unit and integration tests during transition -- test written using dockertest DX may be a bit worse +* some test logic duplication between unit and integration tests during transition +* test written using dockertest DX may be a bit worse ### Neutral -- learning curve for BDD style tests -- some discovery required for e2e transition to dockertest +* learning curve for BDD style tests +* some discovery required for e2e transition to dockertest ## Further Discussions @@ -243,8 +243,8 @@ It may be useful if test suites could be run in integration mode (with mocked te with e2e fixtures (with real tendermint and many nodes). Integration fixtures could be used for quicker runs, e2e fixures could be used for more battle hardening. -A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847) -is in progress for unit tests demonstrating BDD. +A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847) +is in progress for unit tests demonstrating BDD [Rejected]. Observing that a strength of BDD specifications is their readability, and a con is the cognitive load while writing and maintaining, current consensus is to reserve BDD use for places in the SDK where complex rules and module interactions are demonstrated. diff --git a/docs/building-chain/README.md b/docs/building-chain/README.md new file mode 100644 index 0000000000..141fe5cd40 --- /dev/null +++ b/docs/building-chain/README.md @@ -0,0 +1,12 @@ + + +# Building a Chain + +This repository contains documentation on concepts developers need to know in order to build a Cosmos SDK applications. + +1. [Overview of `app.go` and how to wire it up](./app-go.md) +2. [Dependency Injection in the SDK](./depinject.md) diff --git a/docs/building-chain/app-go.md b/docs/building-chain/app-go.md new file mode 100644 index 0000000000..2d5f6dc85c --- /dev/null +++ b/docs/building-chain/app-go.md @@ -0,0 +1,24 @@ + + +# Overview of `app.go` and how to wire it up + +This section is intended to provide an overview of the `app.go` file and is still a work in progress. +For now we invite you to read the [tutorials](https://tutorials.cosmos.network) for a deep dive on how to build a chain. + + \ No newline at end of file diff --git a/docs/building-chain/depinject.md b/docs/building-chain/depinject.md new file mode 100644 index 0000000000..6f1c45f0dc --- /dev/null +++ b/docs/building-chain/depinject.md @@ -0,0 +1,17 @@ + + +# Dependency Injection + +This section is intended to provide an overview of the `depinject` package and is still a work in progress. +The SDK uses a dependency injection framework called `depinject` for helping building a chain faster. + +## `AppConfig` + +* https://pkg.go.dev/cosmossdk.io/core/appconfig + + +## `depinject` + +* https://pkg.go.dev/cosmossdk.io/depinject diff --git a/docs/building-modules/simulator.md b/docs/building-modules/simulator.md index 91145a87f3..779f053034 100644 --- a/docs/building-modules/simulator.md +++ b/docs/building-modules/simulator.md @@ -1,3 +1,7 @@ + + # Module Simulation ## Prerequisites diff --git a/docs/building-modules/testing.md b/docs/building-modules/testing.md new file mode 100644 index 0000000000..20edb229da --- /dev/null +++ b/docs/building-modules/testing.md @@ -0,0 +1,105 @@ + + +# Module & App Testing + +The Cosmos SDK contains different types of [tests](https://martinfowler.com/articles/practical-test-pyramid.html). +These tests have different goals and are used at different stages of the development cycle. +We advice, as a general rule, to use tests at all stages of the development cycle. +It is adviced, as a chain developer, to test your application and modules in a similar way than the SDK. + +The rationale behind testing can be found in [ADR-59](https://docs.cosmos.network/main/architecture/adr-059-test-scopes.html). + +## Unit Tests + +Unit tests are the lowest test category of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). +All packages and modules should have unit test coverage. Modules should have their dependencies mocked: this means mocking keepers. + +The SDK uses `mockgen` to generate mocks for keepers: + ++++ https://github.com/cosmos/cosmos-sdk/blob/dd556936b23d7443cb7fb1da394c35117efa9da7/scripts/mockgen.sh#L29 + +You can read more about mockgen [here](https://github.com/golang/mock). + +### Example + +As an example, we will walkthrough the [keeper tests](https://github.com/cosmos/cosmos-sdk/blob/a92c291880eb6240b7221173282fee0c5f2adb05/x/gov/keeper/keeper_test.go) of the `x/gov` module. + +The `x/gov` module has a `Keeper` type requires a few external dependencies (ie. imports outside `x/gov` to work properly). + ++++ https://github.com/cosmos/cosmos-sdk/blob/a92c291880eb6240b7221173282fee0c5f2adb05/x/gov/keeper/keeper.go#L64-L68 + +In order to only test `x/gov`, we mock the [expected keepers](https://docs.cosmos.network/v0.46/building-modules/keeper.html#type-definition) and instantiate the `Keeper` with the mocked dependencies. Note that we may need to configure the mocked dependencies to return the expected values: + ++++ https://github.com/cosmos/cosmos-sdk/blob/a92c291880eb6240b7221173282fee0c5f2adb05/x/gov/keeper/common_test.go#L66-L83 + +This allows us to test the `x/gov` module without having to import other modules. + ++++ https://github.com/cosmos/cosmos-sdk/blob/a92c291880eb6240b7221173282fee0c5f2adb05/x/gov/keeper/keeper_test.go#L3-L35 + +We can test then create unit tests using the newly created `Keeper` instance. + ++++ https://github.com/cosmos/cosmos-sdk/blob/a92c291880eb6240b7221173282fee0c5f2adb05/x/gov/keeper/keeper_test.go#L73-L91 + +## Integration Tests + +Integration tests are at the second level of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). +In the SDK, we locate our integration tests under [`/tests/integrations`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/integration). + +The goal of these integration tests is to test a component with a minimal application (i.e. not `simapp`). The minimal application is defined with the help of [`depinject`](../building-chain/depinject.md) – the SDK dependency injection framework, and includes all necessary modules to test the component. With the helps of the SDK testing package, we can easily create a minimal application and start the application with a set of genesis transactions: . + +### Example + +Here, we will walkthrough the integration tests of the `x/distribution` module. The `x/distribution` module has, in addition to keeper unit tests, integration tests that test the `x/distribution` module with a minimal application. This is expected as you may want to test the `x/distribution` module with actual application logic, instead of only mocked dependencies. + +For creating a minimal application, we use [`simtestutil.Setup`](https://github.com/cosmos/cosmos-sdk/blob/main/testutil/sims/app_helpers.go#L98-L102) and an [`AppConfig`](../building-chain/depinject.md) of the `x/distribution` minimal dependencies. + +For instance, the `AppConfig` of `x/distribution` is defined as: + +* https://github.com/cosmos/cosmos-sdk/blob/main/x/distribution/testutil/app_config.go + +This is a stripped down version of the `simapp` `AppConfig`: + +* https://github.com/cosmos/cosmos-sdk/blob/main/simapp/app_config.go + +_Note, you can as well use the `AppConfig` configurator for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/15b04c2a87e433fe97877a32162b96ba2ebf8982/x/slashing/app_test.go#L54-L61). There no difference between those two ways, use whichever you prefer._ + ++++ https://github.com/cosmos/cosmos-sdk/blob/e09516f4795c637ab12b30bf732ce5d86da78424/tests/integration/distribution/keeper/keeper_test.go#L28-L33 + +Now the types are injected and we can use them for our tests: + ++++ https://github.com/cosmos/cosmos-sdk/blob/e09516f4795c637ab12b30bf732ce5d86da78424/tests/integration/distribution/keeper/keeper_test.go#L21-L53 + +## Simulations + +Simulations uses as well a minimal application, built with [`depinject`](../building-chain/depinject.md): + +Following is an example for `x/gov/` simulations: + ++++ https://github.com/cosmos/cosmos-sdk/blob/0fbcb0b18381d19b7e556ed07e5467129678d68d/x/gov/simulation/operations_test.go#L290-L307 + ++++ https://github.com/cosmos/cosmos-sdk/blob/main/x/gov/simulation/operations_test.go#L67-L109 + +## End-to-end Tests + +End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). +They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e). + +For that, the SDK is using `simapp` but you should use your own application (`appd`). +Here are some examples: + +* SDK E2E tests: . +* Cosmos Hub E2E tests: . +* Osmosis E2E tests: . + +_Note, the SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/architecture/adr-059-test-scopes.html). This page will eventually be updated with better examples._ + +## Summary + +| Scope | App Fixture | Mocks? | +| ----------- | ----------- | ------ | +| Unit | None | Yes | +| Integration | `depinject` | Some | +| Simulation | `depinject` | No | +| E2E | `appd` | No | diff --git a/docs/cosmwasm/README.md b/docs/cosmwasm/README.md index 409f31ebda..b774d9d99d 100644 --- a/docs/cosmwasm/README.md +++ b/docs/cosmwasm/README.md @@ -1,7 +1,7 @@ # CosmWasm smart contracts diff --git a/docs/ibc/README.md b/docs/ibc/README.md index 872e5c0dcf..a895a39e23 100644 --- a/docs/ibc/README.md +++ b/docs/ibc/README.md @@ -1,7 +1,7 @@ # IBC diff --git a/docs/migrations/README.md b/docs/migrations/README.md index 373dbd398c..8773122079 100644 --- a/docs/migrations/README.md +++ b/docs/migrations/README.md @@ -1,13 +1,13 @@ # Migrations This document contains all the migration guides to update your app and modules to the current Cosmos SDK. -1. Chain Upgrade Guide to v0.46: - * See [UPGRADING.md (TODO)](https://github.com/cosmos/cosmos-sdk/blob/main/UPGRADING.md) for breaking changes and deprecations upgrade instructions. - * See [Release Notes](https://github.com/cosmos/cosmos-sdk/blob/v0.46.0/RELEASE_NOTES.md) and [changelog](https://github.com/cosmos/cosmos-sdk/blob/v0.46.0/CHANGELOG.md) for the exhaustive list of API and State Machine breaking changes. +1. Chain Upgrade Guide to v0.47: + * See [UPGRADING.md](https://github.com/cosmos/cosmos-sdk/blob/main/UPGRADING.md) for breaking changes and deprecations upgrade instructions. + * See [Release Notes](https://github.com/cosmos/cosmos-sdk/blob/main/RELEASE_NOTES.md) and [changelog](https://github.com/cosmos/cosmos-sdk/blob/main/CHANGELOG.md) for the exhaustive list of API and State Machine breaking changes. diff --git a/docs/run-node/README.md b/docs/run-node/README.md index ec6eca8bd5..36d67258b1 100644 --- a/docs/run-node/README.md +++ b/docs/run-node/README.md @@ -1,7 +1,7 @@ # Running a Node, API and CLI diff --git a/x/simulation/decoder_test.go b/x/distribution/simulation/decoder_test.go similarity index 100% rename from x/simulation/decoder_test.go rename to x/distribution/simulation/decoder_test.go diff --git a/x/simulation/operations_test.go b/x/distribution/simulation/operations_test.go similarity index 100% rename from x/simulation/operations_test.go rename to x/distribution/simulation/operations_test.go