Steps to Reproduce:
1. Setup the pool.
2. Run small load test.
3. Keep pool without any load for long time (~10 hours).
4. Run load test:
=> Load test was failed because of TOK-425, nodes were crashed.
5. Run one more load test with NYMs only (without fees).
=> NACKs were received for each request because of View Change, part of the nodes were failed because of the same reason.
36 View Changes were happened during 10 minutes right after start of load test.
Nodes 3 and 4 had ViewNo 0 after 27 and 28.
Node 7 had ViewNo 33.
Nodes 2, 5, 6, 8, 9 had ViewNo 35.
The rest ones stopped view changing on ViewNo 36.
Logs and metrics: s3://qanodelogs/indy-1696/NodeXX/
To get logs, run following command on log processor machine:
aws s3 cp --recursive s3://qanodelogs/indy-1696/ /home/ev/logs/indy-1696/
Check whether this is a known issue or not. Most probably this ticket is related to plugins throwing exceptions
INDY-1698 and cyclic view change issues.