Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1429

Perf pool stopped ordering txns during load test

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
    • Sprint:
      EV 18.13 Benchmark hardening

      Description

      Versions:
      indy-node: 1.4.478
      indy-plenum: 1.4.418

      Node config
      logLevel=DEBUG
      forceViewChangeFreq=600

      What happened
      When running load test on with two agents running

      python3.5 perf_processes.py -c 20 -n 1000 -k nym -g pool_transactions_genesis
      

      in a loop after some view change pool stopped ordering transactions. Ledgers were NOT corrupted, all nodes eventually reached same state, but refused to order any new txn. Restarting all nodes in a pool fixed problem temporarily.

      After investigation of log files it turned out that in the beginning of some view primary generated first PREPREPARE with more than 9000 txns, with size about 640k, while default MSG_LEN_LIMIT is just 128k. So, this PREPREPARE was not delivered to other nodes and no new transactions were ordered.

      It's recommended to either increase MSG_LEN_LIMIT to 768k or decrease Max3PCBatchSize accordingly.

      Acceptance criteria
      Find the root cause and create a ticket for fixing

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              sergey.khoroshavin Sergey Khoroshavin
              Reporter:
              sergey.khoroshavin Sergey Khoroshavin
              Watchers:
              Sergey Khoroshavin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: