Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1893

Validator stuck in continual view change

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.6.83
    • Component/s: None
    • Labels:
      None
    • Environment:

      MainNet, indy-node 1.6.78

    • Documentation Impact:
      Unset
    • Epic Link:
    • Sprint:
      EV 18.24

      Description

      One of the steward validators on MainNet (indy-node 1.6.78) has been stuck in a view change for an extended time. This first came to notice when attempting to retrieve its status using get-validator-info from a client:

      pool(mainnet):wallet(mainnet_wallet):did(J4N...Y5q):indy> ledger get-validator-info nodes=NewtonD
      Validator Info:
      Transaction has been rejected: Client request is discarded since view change is in progress
      {}
      

      This is the only node that experienced this issue. All other nodes view and primary information was consistant. A transaction was posted to the ledger, and this node did not post it in its database.

      The node was able to recover following a restart of the service. Please examine the attached logs and validator-info dump (before the restart) to see if there is information that can be used to improved the stability of the product.

      UPDATE:

      At about 10:45 MST 6 nodes on TestNet owned by Evernym were rebooted simultaneously to apply a kernel update. One of these nodes was the current primary node. As a result of this, a view change was initiated, which was not completed properly on all nodes. 2 of the Evernym nodes selected a different primary than the rest, and 10 of the nodes owned by other stewards got into the same state that we see in this ticket: a never-completed view change.

      pool(testnet):wallet(testnet_wallet):did(6fe...zSJ):indy> ledger get-validator-info
      Validator Info:
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      Transaction has been rejected: Client request is discarded since view change is in progress
      ...

      I will be attaching logs from all of the nodes that I can get my hands on.

      Restarting effected nodes clears the issue, but takes a lot of time since it must be done manually by our dispersed stewards. (see INDY-1896) This causes much concern on TestNet these days. Updating the priority accordingly.

       

        Attachments

        1. australia.tgz
          2.46 MB
        2. brazil.tgz
          2.41 MB
        3. canada.tgz
          1.45 MB
        4. england.tgz
          4.14 MB
        5. korea.tgz
          3.57 MB
        6. NewtonD_20181203.tar.gz
          7.73 MB
        7. NewtonD_config_before
          21 kB
        8. NodeTwinPeek.tar.gz
          2.08 MB
        9. singapore.tgz
          3.79 MB
        10. validator-info.log-20181207.gz
          64 kB
        11. virginia.tgz
          5.33 MB
        12. xsvalidatorec2irl.log.tgz
          2.38 MB

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mgbailey Mike Bailey
              Watchers:
              Alexander Shcherbakov, Artem Obruchnikov, Mike Bailey
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: