Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1256

STN lost consensus

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • Highest
    • Resolution: Done
    • None
    • None
    • None
    • None
    • STN, running 1.3.55

    • 18.07 Stability & Monitoring, EV 18.09 Stability-RocksDB, 18.08 Stability-Monitoring

    Description

      We lost the ability to post transactions to the STN ledger.  Looking in the logs, a view change, cause unknown, occurred on all the nodes that I have access to at March 30, 02:42 GMT.  The last transaction posted to the ledger is timestamped 1522356428, which is March 29, 20:47. I have access to 7 of 10 nodes, and their logs are attached. 

      At 17:16 GMT I restarted the indy-node service on those nodes and saw that in the view change the original primary was restored, and I was again able to post transactions. 

      Please analyze the attached logs, and determine insofar as is possible:

      • What caused the view change?
      • Why were we unable to post transactions?
      • Following the restart, were all 10 of the nodes participating in consensus?
        • The nodes that did not restart are NewtonD, RFCU, and ibm

      Secondary questions:

      • Were there attempts to post transactions to the ledger before the view change that failed?
      • When did the fault state begin, where transactions were not posting?
      • Did the view change cause the lack of consensus, and why would it?

      Attachments

        1. can-stn-p001_log_20180411.tgz
          1.05 MB
        2. can-stn-p001-post.tar
          880 kB
        3. domain_ledger.tar
          47 kB
        4. live_system_upgrade_problem.7z
          543 kB
        5. lon-stn-p001_log_20180411.tgz
          1.06 MB
        6. lon-stn-p001-post.tar
          881 kB
        7. nva-stn-p001_log_20180411.tgz
          1.06 MB
        8. nva-stn-p001-post.tar
          888 kB
        9. pool_ledger.tgz
          6 kB
        10. sao-stn-p001_log_20180411.tgz
          1.06 MB
        11. sao-stn-p001-post.tar
          947 kB
        12. seo-stn-p001_log_20180411.tgz
          1.06 MB
        13. seo-stn-p001-post.tar
          921 kB
        14. sgp-stn-p001_log_20180411.tgz
          1.07 MB
        15. sgp-stn-p001-post.tar
          928 kB
        16. syd-stn-p001_log_20180411.tgz
          1.43 MB
        17. syd-stn-p001-post.tar
          925 kB

        Issue Links

          Activity

            People

              ozheregelya Olga Zheregelya
              mgbailey Mike Bailey
              Dmitry Surnin, Kelly Wilson, Mike Bailey, NIkita Zhigunenko, Olga Zheregelya, Vladimir Shishkin
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: