Currently PTE keeps running even if transactions fail due to underlying Fabric bug or due to any server side environment issue. There is no value in continuing the run when failures occur. Instead, it will help aborting the test and sending failure report to tester so a bug can be filed sooner than later.
- Add parameter for failure threshold. This could either be % failures or absolute threshold count for # of failed transactions.
2. Check failure threshold value and abort test when threshold is crossed i.e. failure count > threshold.
3. Save logs. This should include test client CPU, memory, disk and block chain network - peer, order and CA container - CPU, memory, disk utilization when threshold is crossed. test client stats required so we can isolate if it's client side issue or network side issue.
4. Generate report to show total transactions, total successful transaction, total failed transactions, test name, network id (applicable for IBP), type of network (IBP - starter, EP. or simple non-IBP, if it can be determined).
At a more basic level, pte should abort when return non-zero exit codes for each step failure (e.g. instantiation).
Related: add option to operator/testInput to allow testers to specify threshhold.