Details
High Description
Environment:
indy-node 1.6.598
Steps to Reproduce:
1. Setup the pool.
2. Run small load test.
3. Keep pool without any load for long time (~10 hours).
4. Run load test:
python3.5 perf_processes.py -g ~/ext_transactions_genesis -m t -n 1 -c 10 -l 1 -y freeflow -k "[{\"fees_nym\":{\"count\": 4, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}, {\"fees_schema\":{\"count\": 1, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}, {\"attrib\":{\"count\": 3}}, {\"cred_def\":{\"count\": 1}}, {\"revoc_reg_def\":{\"count\": 1}}, {\"revoc_reg_entry\":{\"count\": 1}}, {\"payment\":{\"count\": 9, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}]" -b 200
=> Load test was failed because of TOK-425, nodes were crashed.
5. Run one more load test with NYMs only (without fees).
=> NACKs were received for each request because of View Change, part of the nodes were failed because of the same reason.
Actual Results:
36 View Changes were happened during 10 minutes right after start of load test.
Nodes 3 and 4 had ViewNo 0 after 27 and 28.
Node 7 had ViewNo 33.
Nodes 2, 5, 6, 8, 9 had ViewNo 35.
The rest ones stopped view changing on ViewNo 36.
Logs and metrics: s3://qanodelogs/indy-1696/NodeXX/
To get logs, run following command on log processor machine:
aws s3 cp --recursive s3://qanodelogs/indy-1696/ /home/ev/logs/indy-1696/
Acceptance criteria:
Check whether this is a known issue or not. Most probably this ticket is related to plugins throwing exceptions INDY-1698 and cyclic view change issues.