Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1183

Newly upgraded STN fails to accept transactions

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      STN running 1.3.52

    • Sprint:
      Sprint 18.04

      Description

      Today we upgraded the STN from 1.2.50 to 1.3.52. We were posting transactions to it successfully yesterday before the upgrade, but now we are unable to successfully post transactions.  For example, at 19:17:17 I attempted to post a DID:

      indy@sandbox> send NYM dest=SYLcHfW7PEU74ZWHkonbqx verkey=~7Xz2uHvQ8swWbchnsj1aAN
      Adding nym SYLcHfW7PEU74ZWHkonbqx
      

      But it did not get written to the ledger.

      The upgrade was successful for all nodes except for ibm, where we performed a manual upgrade.  I have access to 7 of 10 nodes, whose logs are attached.  All of our nodes (plus ibm) show that the same primaries have been selected on them.  validator-info shows connectivity between all nodes.

      One thing that I am seeing are errors regarding consistency proofs:

      2018-02-20 19:15:21,880 | ERROR    | ledger_manager.py    (1001) | _buildConsistencyProof | virginia cannot build consistency proof till 35 since its ledger size is 32
      

      It appears that one of the nodes, pcValidator01, has 3 more transactions than the rest?  I will be contacting that steward for logs and ledger data, which I will post here.

      If my supposition that there are 3 additional transactions in one validator's ledger is correct, then there are 2 things to look at: 

      1. How did the transactions get in there?
      2. Why does one node's ledger being out of sync prevent transactions in the whole pool?

        Attachments

        1. can-stn-p001_20180220.tgz
          2.20 MB
        2. config_ledger
          9 kB
        3. domain_ledger
          7 kB
        4. lon-stn-p001_20180220.tgz
          2.73 MB
        5. nva-stn-p001_20180220.tgz
          2.26 MB
        6. pcvalidator01_logs_2018_02_20.zip
          52 kB
        7. pool_ledger
          8 kB
        8. sao-stn-p001_20180220.tgz
          2.76 MB
        9. seo-stn-p001_20180220.tgz
          2.68 MB
        10. sgp-stn-p001_20180220.tgz
          2.69 MB
        11. syd-stn-p001_20180220.tgz
          2.26 MB

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mgbailey Mike Bailey
              Watchers:
              Alexander Shcherbakov, Mike Bailey, Olga Zheregelya
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: