Details
Description
Step to reproduce:
1) setup pool with 25 nodes on AWS
2) set 1,06kk txns to ledger
3) run concurrent load test:
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 286 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_schema\": {\"file_name\": \"./load_schema/successful\"}}"
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_attrib\": {\"file_name\": \"./load_attrib/successful\"}}"
python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_nym\": {\"file_name\": \"./load_nym/successful\"}}"
Actual results:
2018-07-05 07:16:58,838 | INFO | replica.py ( 884) | dispatchThreePhaseMsg | REPLICA:(Node18:5) Node18:5 stashing 3 phase message PREPREPARE{'discarded': 1, 'digest': '28eb27c43db6b7472b72e9fa547f7fc7e7f1eb8e6d4fc521c225a98c5fc3ca91', 'reqIdr': ['2f051e9054ea521e0acce5949839e3a3f174ba6bc063974b6fb796f04564dc34'], 'ppSeqNo': 21116, 'instId': 5, 'ppTime': 1530775018, 'ledgerId': 1, 'txnRootHash': None, 'stateRootHash': None, 'viewNo': 3} since ppSeqNo 21116 is not between 0 and 300)
Workaround:
Restart whole pool via CLI
Logs are available on LogProcessor