docs: Add initial doc for system tests (#22002)
This commit is contained in:
parent
0ea2445acf
commit
5c4f4ac305
58
docs/build/building-apps/06-system-tests.md
vendored
Normal file
58
docs/build/building-apps/06-system-tests.md
vendored
Normal file
@ -0,0 +1,58 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# System Tests
|
||||
|
||||
System tests provide a framework to write and execute black box tests against a running chain. This adds another level
|
||||
of confidence on top of unit, integration, and simulations tests, ensuring that business-critical scenarios
|
||||
(like double signing prevention) or scenarios that can't be tested otherwise (like a chain upgrade) are covered.
|
||||
|
||||
## Vanilla Go for Flow Control
|
||||
|
||||
System tests are vanilla Go tests that interact with the compiled chain binary. The `test runner` component starts a
|
||||
local testnet of 4 nodes (by default) and provides convenient helper methods for accessing the
|
||||
`system under test (SUT)`.
|
||||
A `CLI wrapper` makes it easy to access keys, submit transactions, or execute operations. Together, these components
|
||||
enable the replication and validation of complex business scenarios.
|
||||
|
||||
Here's an example of a double signing test, where a new node is added with the same key as the first validator:
|
||||
[double signing test example](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/fraud_test.go)
|
||||
|
||||
The [getting started tutorial](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/getting_started.md)
|
||||
contains a step-by-step guide to building and running your first system test. It covers setting chain state via genesis
|
||||
or
|
||||
transactions and validation via transaction response or queries.
|
||||
|
||||
## Design Principles and Guidelines
|
||||
|
||||
System tests are slower compared to unit or integration tests as they interact with a running chain. Therefore, certain
|
||||
principles can guide their usage:
|
||||
|
||||
- **Perspective:** Tests should mimic a human interacting with the chain from the outside. Initial states can be set via
|
||||
genesis or transactions to support a test scenario.
|
||||
- **Roles:** The user can have multiple roles such as validator, delegator, granter, or group admin.
|
||||
- **Focus:** Tests should concentrate on happy paths or business-critical workflows. Unit and integration tests are
|
||||
better suited for more fine-grained testing.
|
||||
- **Workflows:** Test workflows and scenarios, not individual units. Given the high setup costs, it is reasonable to
|
||||
combine multiple steps and assertions in a single test method.
|
||||
- **Genesis Mods:** Genesis modifications can incur additional time costs for resetting dirty states. Reuse existing
|
||||
accounts (node0..n) whenever possible.
|
||||
- **Framework:** Continuously improve the framework for better readability and reusability.
|
||||
|
||||
## Errors and Debugging
|
||||
|
||||
All output is logged to `systemtests/testnet/node{0..n}.out`. Usually, `node0.out` is very noisy as it receives the CLI
|
||||
connections. Prefer any other node's log to find stack traces or error messages.
|
||||
|
||||
Using system tests for state setup during debugging has become very handy:
|
||||
|
||||
- Start the test with one node only and verbose output:
|
||||
|
||||
```sh
|
||||
go test -v -tags=system_test ./ --run TestAccountCreation --verbose --nodes-count=1
|
||||
```
|
||||
|
||||
- Copy the CLI command for the transaction and modify the test to stop before the command
|
||||
- Start the node with `--home=<project-home>/tests/systemtests/testnet/node0/<binary-name>/` in debug mode
|
||||
- Execute CLI command from shell and enter breakpoints
|
||||
30
docs/build/building-modules/16-testing.md
vendored
30
docs/build/building-modules/16-testing.md
vendored
@ -86,38 +86,26 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/tests/integration/bank
|
||||
|
||||
## Simulations
|
||||
|
||||
Simulations uses as well a minimal application, built with [`depinject`](../packages/01-depinject.md):
|
||||
Simulations fuzz tests for deterministic message execution. They use a minimal application, built with [`depinject`](../packages/01-depinject.md):
|
||||
|
||||
:::note
|
||||
You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer.
|
||||
Simulations have been refactored to message factories
|
||||
:::
|
||||
|
||||
Following is an example for `x/gov/` simulations:
|
||||
An example for `x/bank/` simulations:
|
||||
|
||||
```go reference
|
||||
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L406-L430
|
||||
https://github.com/cosmos/cosmos-sdk/blob/release/v0.52.x/x/bank/simulation/msg_factory.go#L13-L20
|
||||
```
|
||||
|
||||
```go reference
|
||||
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L90-L132
|
||||
```
|
||||
## System Tests
|
||||
|
||||
## End-to-end Tests
|
||||
System tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
|
||||
They test the whole application flow as black box, from the user perspective. They are located under [`/tests/systemtests`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/systemtests).
|
||||
|
||||
End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
|
||||
They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e).
|
||||
For that, the SDK is using the `simapp` binary, but you should use your own binary.
|
||||
More details about system test can be found in [building-apps](https://docs.cosmos.network/main/build/building-apps/06-system-tests.md)
|
||||
|
||||
<!-- @julienrbrt: makes more sense to use an app wired app to have 0 simapp dependencies -->
|
||||
For that, the SDK is using `simapp` but you should use your own application (`appd`).
|
||||
Here are some examples:
|
||||
|
||||
* SDK E2E tests: <https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e>.
|
||||
* Cosmos Hub E2E tests: <https://github.com/cosmos/gaia/tree/main/tests/e2e>.
|
||||
* Osmosis E2E tests: <https://github.com/osmosis-labs/osmosis/tree/main/tests/e2e>.
|
||||
|
||||
:::note warning
|
||||
The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/build/architecture/adr-059-test-scopes). This page will eventually be updated with better examples.
|
||||
:::
|
||||
|
||||
## Learn More
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user