Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1260

Pool stops taking txns at ~178k txns written in ledger

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • High
    • Resolution: Done
    • None
    • None
    • None
    • 18.07 Stability & Monitoring, 18.08 Stability-Monitoring, EV 18.11 Stability/ViewChange

    Description

      Build Info:
      indy-node 1.3.357

      Steps to Reproduce:
      1. Run multiple load tests (reading and writing) against 25 nodes pool.
      2. Reach ~178k txns written.
      3. Try to write some additional txns (via load script or CLI).
      4. Check validator-info at all nodes.

      Actual Results:
      One of nodes (24) stopped at 49k txns and doesn't catch up. One of nodes (11) stopped at 123k txns and doesn't catch up. All other nodes have 175..178k txns. All nodes show 25/25 reachable hosts in validator-info but pool has no consensus. Whole pool restart doesn't help.
      Journalctl is in attachment. Nodes' logs will be added to google drive.

      Acceptance Criteria:
      Diagnose the issue, decide on a Plan of Attack, and raise the appropriate epics and stories that can be scheduled.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              VladimirWork Vladimir Shishkin
              Alexander Shcherbakov, Artem Obruchnikov, Dmitry Surnin, NIkita Zhigunenko, Vladimir Shishkin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: