Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1562

Part of the nodes are lagged after high load

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.6.73
    • Component/s: None
    • Labels:
      None
    • Documentation Impact:
      Unset
    • Epic Link:
    • Sprint:
      EV 18.16 Releasing 1.6

      Description

      Environment:
      indy-node 1.5.543
      libindy 1.6.1~655
      AWS pool of 25 nodes with 32Gb RAM

      Steps to Reproduce:
      1. Run load for INDY-1388:

      2 clients:>>> python3.5 perf_processes.py -m t -n 1 -t 0.0001 -c 500 -g ~/stab_transactions_genesis_new -k "[{\"nym\": {\"count\": 4}}, {\"schema\":{\"count\": 1}}, {\"attrib\":{\"count\": 3}}, {\"cred_def\":{\"count\": 1}}]"
      
      8 clients:>>> python3.5 perf_processes.py -m t -n 1 -t 0.0001 -c 1125 -g ~/stab_transactions_genesis_new -k "[{\"get_nym\": {\"count\": 9}}, {\"get_attrib\":{\"count\": 2}}, {\"get_schema\":{\"count\": 2}}, {\"get_cred_def\":{\"count\": 2}}]"

      => Reading test was running ~2 hours, writing one was running ~30 minutes.
      2. After end of the test wait for processing of all written txns (more than 2-3 hours).
      => All nodes exclude Node16 wrote 127595 txns, Node 16 wrote 127490 txns.
      3. Write 1 txn.
      => All nodes wrote this txn exclude Node13, Node15 and Node16.
      4. Run small load test with medium load.
      => 190 txns were written by all nodes exclude Node13, Node15 and Node16. 

      Actual Results:
      Nodes 13, 15 and 16 are not writing after load test. Note that in 5 hours after end of step 4 there are no updates. Nodes are in the same state.

      Logs and metrics:
      s3://qanodelogs/indy-1562/NodeXX/
      To get logs, run following command on log processor machine:
      aws s3 cp --recursive s3://qanodelogs/indy-1562/ /home/ev/logs/indy-1562/

        Attachments

          Activity

            People

            Assignee:
            anikitinDSR Andrew Nikitin
            Reporter:
            ozheregelya Olga Zheregelya
            Watchers:
            Andrew Nikitin, Olga Zheregelya
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: