feat: proving: Introduce manual sector fault recovery (#9144)

* rebase

* fix lint error

* fix errors

* add itest

* make gen after rebase

* apply suggestion from review

* make gen after latest rebase

* gen after rebase

* fix waitgroup

* change as per review

* refactor Recovery Batches

* fix CLI

* gen after rebase

* close the channel

documentation/en/api-v0-methods-miner.md

@@ -106,6 +106,8 @@
   * [PiecesListPieces](#PiecesListPieces)
 * [Pledge](#Pledge)
   * [PledgeSector](#PledgeSector)
+* [Recover](#Recover)
+  * [RecoverFault](#RecoverFault)
 * [Return](#Return)
   * [ReturnAddPiece](#ReturnAddPiece)
   * [ReturnDataCid](#ReturnDataCid)
@@ -2265,6 +2267,36 @@ Response:
 }
 ```
 
+## Recover
+
+
+### RecoverFault
+RecoverFault can be used to declare recoveries manually. It sends messages
+to the miner actor with details of recovered sectors and returns the CID of messages. It honors the
+maxPartitionsPerRecoveryMessage from the config
+
+
+Perms: admin
+
+Inputs:
+```json
+[
+  [
+    123,
+    124
+  ]
+]
+```
+
+Response:
+```json
+[
+  {
+    "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"
+  }
+]
+```
+
 ## Return
 
 

documentation/en/cli-lotus-miner.md

@@ -2093,14 +2093,15 @@ USAGE:
    lotus-miner proving command [command options] [arguments...]
 
 COMMANDS:
-   info       View current state information
-   deadlines  View the current proving period deadlines information
-   deadline   View the current proving period deadline information by its index
-   faults     View the currently known proving faulty sectors information
-   check      Check sectors provable
-   workers    list workers
-   compute    Compute simulated proving tasks
-   help, h    Shows a list of commands or help for one command
+   info            View current state information
+   deadlines       View the current proving period deadlines information
+   deadline        View the current proving period deadline information by its index
+   faults          View the currently known proving faulty sectors information
+   check           Check sectors provable
+   workers         list workers
+   compute         Compute simulated proving tasks
+   recover-faults  Manually recovers faulty sectors on chain
+   help, h         Shows a list of commands or help for one command
 
 OPTIONS:
    --help, -h  show help (default: false)
@@ -2210,6 +2211,19 @@ OPTIONS:
 ```
 ```
 
+### lotus-miner proving recover-faults
+```
+NAME:
+   lotus-miner proving recover-faults - Manually recovers faulty sectors on chain
+
+USAGE:
+   lotus-miner proving recover-faults [command options] <faulty sectors>
+
+OPTIONS:
+   --confidence value  number of block confirmations to wait for (default: 5)
+   
+```
+
 ## lotus-miner storage
 ```
 NAME: