Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1360

Out of memory during non-completed viewChange process (under load)

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • Medium
    • Resolution: Done
    • None
    • 1.5
    • None
    • indy-anoncreds 1.0.32
      indy-cli 1.4.0~523
      indy-node 1.3.422
      indy-plenum 1.2.367
      libindy 1.4.0~523
      libindy-crypto 0.4.0
      python3-indy-crypto 0.4.1

    • EV 18.13 Benchmark hardening

    Description

      Prehistory:
      1) use AWS Perfomance pool
      2) create new pool with 20 nodes
      3) write about 1k txns
      4) demote node2 and node3
      5) write about 97k txns

      After all this start the load test
      Used load settings: sudo python3.5 perf_processes.py -g pool_transactions_genesis_performance -c 100 -n 5000 (from single agent)

      Actual results:

      • Pool stops to write after ~139k ledger size
      • Many nodes have been blacklisted (the first time it happened an hour after load start)
      • Current viewNo: 411 (in progress), but we have "INSTANCE_CHANGE {'viewNo': 959, 'reason': 28}

        " in validator-info output

      • From some point "cannot allocate memory" began to appear every ~10 minutes

      Logs will be attached later

      Attachments

        Activity

          People

            sergey.khoroshavin Sergey Khoroshavin
            zhigunenko.dsr NIkita Zhigunenko
            Andrew Nikitin, NIkita Zhigunenko, Sergey Khoroshavin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: