Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1260

Pool stops taking txns at ~178k txns written in ledger

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Link:
    • Sprint:
      18.07 Stability & Monitoring, 18.08 Stability-Monitoring, EV 18.11 Stability/ViewChange

      Description

      Build Info:
      indy-node 1.3.357

      Steps to Reproduce:
      1. Run multiple load tests (reading and writing) against 25 nodes pool.
      2. Reach ~178k txns written.
      3. Try to write some additional txns (via load script or CLI).
      4. Check validator-info at all nodes.

      Actual Results:
      One of nodes (24) stopped at 49k txns and doesn't catch up. One of nodes (11) stopped at 123k txns and doesn't catch up. All other nodes have 175..178k txns. All nodes show 25/25 reachable hosts in validator-info but pool has no consensus. Whole pool restart doesn't help.
      Journalctl is in attachment. Nodes' logs will be added to google drive.

      Acceptance Criteria:
      Diagnose the issue, decide on a Plan of Attack, and raise the appropriate epics and stories that can be scheduled.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                VladimirWork Vladimir Shishkin
                Watchers:
                Alexander Shcherbakov, Artem Obruchnikov, Dmitry Surnin, NIkita Zhigunenko, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: