Affects Version/s: None
Fix Version/s: None
STN, running 1.3.55
Sprint:18.07 Stability & Monitoring, EV 18.09 Stability-RocksDB, 18.08 Stability-Monitoring
We lost the ability to post transactions to the STN ledger. Looking in the logs, a view change, cause unknown, occurred on all the nodes that I have access to at March 30, 02:42 GMT. The last transaction posted to the ledger is timestamped 1522356428, which is March 29, 20:47. I have access to 7 of 10 nodes, and their logs are attached.
At 17:16 GMT I restarted the indy-node service on those nodes and saw that in the view change the original primary was restored, and I was again able to post transactions.
Please analyze the attached logs, and determine insofar as is possible:
- What caused the view change?
- Why were we unable to post transactions?
- Following the restart, were all 10 of the nodes participating in consensus?
- The nodes that did not restart are NewtonD, RFCU, and ibm
- Were there attempts to post transactions to the ledger before the view change that failed?
- When did the fault state begin, where transactions were not posting?
- Did the view change cause the lack of consensus, and why would it?