Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1360

Out of memory during non-completed viewChange process (under load)

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
    • Environment:

      indy-anoncreds 1.0.32
      indy-cli 1.4.0~523
      indy-node 1.3.422
      indy-plenum 1.2.367
      libindy 1.4.0~523
      libindy-crypto 0.4.0
      python3-indy-crypto 0.4.1

    • Epic Link:
    • Sprint:
      EV 18.13 Benchmark hardening

      Description

      Prehistory:
      1) use AWS Perfomance pool
      2) create new pool with 20 nodes
      3) write about 1k txns
      4) demote node2 and node3
      5) write about 97k txns

      After all this start the load test
      Used load settings: sudo python3.5 perf_processes.py -g pool_transactions_genesis_performance -c 100 -n 5000 (from single agent)

      Actual results:

      • Pool stops to write after ~139k ledger size
      • Many nodes have been blacklisted (the first time it happened an hour after load start)
      • Current viewNo: 411 (in progress), but we have "INSTANCE_CHANGE {'viewNo': 959, 'reason': 28}

        " in validator-info output

      • From some point "cannot allocate memory" began to appear every ~10 minutes

      Logs will be attached later

        Attachments

          Activity

            People

            • Assignee:
              sergey.khoroshavin Sergey Khoroshavin
              Reporter:
              zhigunenko.dsr NIkita Zhigunenko
              Watchers:
              Andrew Nikitin, NIkita Zhigunenko, Sergey Khoroshavin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: