From 0dc490979ffbae581e65c034a1ae4827ee1fb403 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= <raul@protocol.ai>
Date: Mon, 27 Jul 2020 15:36:23 +0100
Subject: [PATCH] readme: explain batch-run of randomized test cases. (#159)

---
 README.md | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/README.md b/README.md
index 23fce81e4..4bd662e24 100644
--- a/README.md
+++ b/README.md
@@ -109,6 +109,40 @@ make
 testground run composition -f ./lotus-soup/_compositions/baseline.toml
 ```
 
+## Batch-running randomised test cases
+
+The Oni testkit supports [range parameters](https://github.com/filecoin-project/oni/blob/master/lotus-soup/testkit/testenv_ranges.go),
+which test cases can use to generate random values, either at the instance level
+(each instance computes a random value within range), or at the run level (one
+instance computes the values, and propagates them to all other instances via the
+sync service).
+
+For example:
+
+```toml
+latency_range   = '["20ms", "500ms"]'
+loss_range      = '[0, 0.2]'
+```
+
+Could pick a random latency between 20ms and 500ms, and a packet loss
+probability between 0 and 0.2. We could apply those values through the
+`netclient.ConfigureNetwork` Testground SDK API.
+
+Randomized range-based parameters are specially interesting when combined with
+batch runs, as it enables Monte Carlo approaches to testing.
+
+The Oni codebase includes a batch test run driver in package `lotus-soup/runner`.
+You can point it at a composition file that uses range parameters and tell it to
+run N iterations of the test:
+
+```shell script
+$ go run ./runner -runs 5 _compositions/net-chaos/latency.toml
+```
+
+This will run the test as many times as instructed, and will place all outputs
+in a temporary directory. You can pass a concrete output directory with
+the `-output` flag. 
+
 ## Catalog
 
 ### Test cases part of `lotus-soup`