Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1941

Sovrin MainNet lost consensus

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.6.83
    • Component/s: None
    • Labels:
    • Documentation Impact:
      Unset
    • Sprint:
      Ev-Node 19.02

      Description

      This morning at 11:34 am MST, an automated process detected that it was no longer able to write a transaction to MainNet (indy-node version 1.6.80). A check of validator-info showed that the majority of the nodes were reporting that no primary was selected, although all nodes had the same transaction counts:

      [all]> show primary
      {
          "Aalto": {
              "Node_info": {
                  "Replicas_status": {
                      "Aalto:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "BIGAWSUSEAST1-001": {
              "Node_info": {
                  "Replicas_status": {
                      "BIGAWSUSEAST1-001:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "DustStorm": {
              "Node_info": {
                  "Replicas_status": {
                      "DustStorm:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "NewtonD": {
              "Node_info": {
                  "Replicas_status": {
                      "NewtonD:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "OASFCU": {
              "Node_info": {
                  "Replicas_status": {
                      "OASFCU:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "ServerVS": {
              "Node_info": {
                  "Replicas_status": {
                      "ServerVS:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "Stuard": {
              "Node_info": {
                  "Replicas_status": {
                      "Stuard:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "TNO": {
              "Node_info": {
                  "Replicas_status": {
                      "TNO:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "VeridiumIDC": {
              "Node_info": {
                  "Replicas_status": {
                      "VeridiumIDC:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "amihan-sovrin": {
              "Node_info": {
                  "Replicas_status": {
                      "amihan-sovrin:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "atbsovrin": {
              "Node_info": {
                  "Replicas_status": {
                      "atbsovrin:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "danube": {
              "Node_info": {
                  "Replicas_status": {
                      "danube:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "digitalbazaar": {
              "Node_info": {
                  "Replicas_status": {
                      "digitalbazaar:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "esatus_AG": {
              "Node_info": {
                  "Replicas_status": {
                      "esatus_AG:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "ev1": {
              "Node_info": {
                  "Replicas_status": {
                      "ev1:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "findentity": {
              "Node_info": {
                  "Replicas_status": {
                      "findentity:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "iRespond": {
              "Node_info": {
                  "Replicas_status": {
                      "iRespond:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "ibm": {
              "Node_info": {
                  "Replicas_status": {
                      "ibm:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "icenode": {
              "Node_info": {
                  "Replicas_status": {
                      "icenode:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "pcValidator01": {
              "Node_info": {
                  "Replicas_status": {
                      "pcValidator01:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "prosovitor": {
              "Node_info": {
                  "Replicas_status": {
                      "prosovitor:0": {
                          "Primary": null
                      }
                  }
              }
          },
          "royal_sovrin": {
              "Node_info": {
                  "Replicas_status": {
                      "royal_sovrin:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "trustscience-validator02": {
              "Node_info": {
                  "Replicas_status": {
                      "trustscience-validator02:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          },
          "zaValidator": {
              "Node_info": {
                  "Replicas_status": {
                      "zaValidator:0": {
                          "Primary": "ev1:0"
                      }
                  }
              }
          }
      }
      

      After getting a Sovrin Foundation trustee online, we were able to post a pool_restart transaction, which resulted in a view change and all nodes selected zaValidator as the new primary.

      We now need to determine why the nodes lost their primary, and fell out of consensus, in order to prevent this behavior going forward. We will be posting logs to this ticket, as we receive them from stewards.

        Attachments

        1. 2019-01-16-digitalbazaar.log.xz
          2.58 MB
        2. esatus_AG.tgz
          3.53 MB
        3. ev1_log.tgz
          133 kB
        4. ev1.log.81.xz
          3.64 MB
        5. TNO_2019_01_14_onwards.log
          5.76 MB
        6. TNO_2019_01_14_onwards.log
          5.76 MB
        7. zaValidator.log.zip
          3.69 MB

          Activity

            People

            Assignee:
            anikitinDSR Andrew Nikitin
            Reporter:
            mgbailey Mike Bailey
            Watchers:
            Andrew Nikitin, Mike Bailey
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: