Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1183

Newly upgraded STN fails to accept transactions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      STN running 1.3.52

    • Sprint:
      Sprint 18.04

      Description

      Today we upgraded the STN from 1.2.50 to 1.3.52. We were posting transactions to it successfully yesterday before the upgrade, but now we are unable to successfully post transactions.  For example, at 19:17:17 I attempted to post a DID:

      indy@sandbox> send NYM dest=SYLcHfW7PEU74ZWHkonbqx verkey=~7Xz2uHvQ8swWbchnsj1aAN
      Adding nym SYLcHfW7PEU74ZWHkonbqx
      

      But it did not get written to the ledger.

      The upgrade was successful for all nodes except for ibm, where we performed a manual upgrade.  I have access to 7 of 10 nodes, whose logs are attached.  All of our nodes (plus ibm) show that the same primaries have been selected on them.  validator-info shows connectivity between all nodes.

      One thing that I am seeing are errors regarding consistency proofs:

      2018-02-20 19:15:21,880 | ERROR    | ledger_manager.py    (1001) | _buildConsistencyProof | virginia cannot build consistency proof till 35 since its ledger size is 32
      

      It appears that one of the nodes, pcValidator01, has 3 more transactions than the rest?  I will be contacting that steward for logs and ledger data, which I will post here.

      If my supposition that there are 3 additional transactions in one validator's ledger is correct, then there are 2 things to look at: 

      1. How did the transactions get in there?
      2. Why does one node's ledger being out of sync prevent transactions in the whole pool?

        Attachments

        1. can-stn-p001_20180220.tgz
          2.20 MB
          Mike Bailey
        2. config_ledger
          9 kB
          Mike Bailey
        3. domain_ledger
          7 kB
          Mike Bailey
        4. lon-stn-p001_20180220.tgz
          2.73 MB
          Mike Bailey
        5. nva-stn-p001_20180220.tgz
          2.26 MB
          Mike Bailey
        6. pool_ledger
          8 kB
          Mike Bailey
        7. sao-stn-p001_20180220.tgz
          2.76 MB
          Mike Bailey
        8. seo-stn-p001_20180220.tgz
          2.68 MB
          Mike Bailey
        9. sgp-stn-p001_20180220.tgz
          2.69 MB
          Mike Bailey
        10. syd-stn-p001_20180220.tgz
          2.26 MB
          Mike Bailey

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            mgbailey Mike Bailey
            Watchers:
            Alexander Shcherbakov, Mike Bailey, Olga Zheregelya
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: