Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      STN, running 1.3.55

    • Epic Link:
    • Sprint:
      18.07 Stability & Monitoring, EV 18.09 Stability-RocksDB, 18.08 Stability-Monitoring

      Description

      We lost the ability to post transactions to the STN ledger.  Looking in the logs, a view change, cause unknown, occurred on all the nodes that I have access to at March 30, 02:42 GMT.  The last transaction posted to the ledger is timestamped 1522356428, which is March 29, 20:47. I have access to 7 of 10 nodes, and their logs are attached. 

      At 17:16 GMT I restarted the indy-node service on those nodes and saw that in the view change the original primary was restored, and I was again able to post transactions. 

      Please analyze the attached logs, and determine insofar as is possible:

      • What caused the view change?
      • Why were we unable to post transactions?
      • Following the restart, were all 10 of the nodes participating in consensus?
        • The nodes that did not restart are NewtonD, RFCU, and ibm

      Secondary questions:

      • Were there attempts to post transactions to the ledger before the view change that failed?
      • When did the fault state begin, where transactions were not posting?
      • Did the view change cause the lack of consensus, and why would it?

        Attachments

        1. can-stn-p001_log_20180411.tgz
          1.05 MB
        2. can-stn-p001-post.tar
          880 kB
        3. domain_ledger.tar
          47 kB
        4. live_system_upgrade_problem.7z
          543 kB
        5. lon-stn-p001_log_20180411.tgz
          1.06 MB
        6. lon-stn-p001-post.tar
          881 kB
        7. nva-stn-p001_log_20180411.tgz
          1.06 MB
        8. nva-stn-p001-post.tar
          888 kB
        9. pool_ledger.tgz
          6 kB
        10. sao-stn-p001_log_20180411.tgz
          1.06 MB
        11. sao-stn-p001-post.tar
          947 kB
        12. seo-stn-p001_log_20180411.tgz
          1.06 MB
        13. seo-stn-p001-post.tar
          921 kB
        14. sgp-stn-p001_log_20180411.tgz
          1.07 MB
        15. sgp-stn-p001-post.tar
          928 kB
        16. syd-stn-p001_log_20180411.tgz
          1.43 MB
        17. syd-stn-p001-post.tar
          925 kB

          Issue Links

            Activity

              People

              • Assignee:
                ozheregelya Olga Zheregelya
                Reporter:
                mgbailey Mike Bailey
                Watchers:
                Dmitry Surnin, Kelly Wilson, Mike Bailey, NIkita Zhigunenko, Olga Zheregelya, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: