Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1696

35 view changes were happened during 10 minutes after nodes failure because of invalid request

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.6.78
    • Component/s: None
    • Labels:
    • Documentation Impact:
      Unset
    • Epic Link:
    • Sprint:
      Ev 18.20

      Description

      Environment:
      indy-node 1.6.598

      Steps to Reproduce:
      1. Setup the pool.
      2. Run small load test.
      3. Keep pool without any load for long time (~10 hours).
      4. Run load test:

      python3.5 perf_processes.py -g ~/ext_transactions_genesis -m t -n 1 -c 10 -l 1 -y freeflow -k "[{\"fees_nym\":{\"count\": 4, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}, {\"fees_schema\":{\"count\": 1, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}, {\"attrib\":{\"count\": 3}}, {\"cred_def\":{\"count\": 1}}, {\"revoc_reg_def\":{\"count\": 1}}, {\"revoc_reg_entry\":{\"count\": 1}}, {\"payment\":{\"count\": 9, \"payment_addrs_count\": 100,\"payment_method\":\"sov\", \"plugin_lib\": \"libsovtoken.so\", \"plugin_init_func\":\"sovtoken_init\"}}]" -b 200

      => Load test was failed because of TOK-425, nodes were crashed.
      5. Run one more load test with NYMs only (without fees).
      => NACKs were received for each request because of View Change, part of the nodes were failed because of the same reason.

      Actual Results:
      36 View Changes were happened during 10 minutes right after start of load test.
      Nodes 3 and 4 had ViewNo 0 after 27 and 28.
      Node 7 had ViewNo 33.
      Nodes 2, 5, 6, 8, 9 had ViewNo 35.
      The rest ones stopped view changing on ViewNo 36.

      Logs and metrics: s3://qanodelogs/indy-1696/NodeXX/
      To get logs, run following command on log processor machine:
      aws s3 cp --recursive s3://qanodelogs/indy-1696/ /home/ev/logs/indy-1696/

      Acceptance criteria:

      Check whether this is a known issue or not. Most probably this ticket is related to plugins throwing exceptions INDY-1698 and cyclic view change issues.

        Attachments

          Activity

            People

            • Assignee:
              zhigunenko.dsr NIkita Zhigunenko
              Reporter:
              ozheregelya Olga Zheregelya
              Watchers:
              NIkita Zhigunenko, Olga Zheregelya, Renata Toktar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: