It appears that view change on pools of 19 nodes or more can cause the pool to stop functioning.
I have a pool with 25 nodes
After adding all the nodes to the pool from a genesis of 7 nodes I had 30 total transactions in the ledger.
I set the load scripts to run transactions every 6 - 30 seconds for an hour.
I have 25 clients across 5 machines each sending 10 transaction bursts every 6 - 30 seconds. Each will send 1400 total transactions which should create activity on the pool for about an hour.
After 524 transaction had been sent the pool stopped taking new transactions. Each node match the other nodes and the log files did not have any errors. I did see in some logs a view change was being requested so I believe this has to do with view change on a large pool.
Logs and Screenshots
- Attached are the smaller logs and the current ones when the nodes failed.
- Also attached is a screenshot of a search I did in the logs. I searched for "proposed_view_change". Only 12 of the 25 nodes have a propose view change and it looks like Node 7 was going to run a different view change ID 991 not ID 992 that the other show.
Some of what I see in the logs around the same time stamp are the following