Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1455

Pool cannot order transactions because Node set incorrect watermarks after it's restart

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      indy-node 1.4.490

    • Epic Link:
    • Sprint:
      EV 18.14 Monitoring/Stability

      Description

      Step to reproduce:
      1) setup pool with 25 nodes on AWS
      2) set 1,06kk txns to ledger
      3) run concurrent load test:

      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 286 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_schema\": {\"file_name\": \"./load_schema/successful\"}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_attrib\": {\"file_name\": \"./load_attrib/successful\"}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_nym\": {\"file_name\": \"./load_nym/successful\"}}"
      

      Actual results:

      2018-07-05 07:16:58,838 | INFO     | replica.py           ( 884) | dispatchThreePhaseMsg | REPLICA:(Node18:5) Node18:5 stashing 3 phase message PREPREPARE{'discarded': 1, 'digest': '28eb27c43db6b7472b72e9fa547f7fc7e7f1eb8e6d4fc521c225a98c5fc3ca91', 'reqIdr': ['2f051e9054ea521e0acce5949839e3a3f174ba6bc063974b6fb796f04564dc34'], 'ppSeqNo': 21116, 'instId': 5, 'ppTime': 1530775018, 'ledgerId': 1, 'txnRootHash': None, 'stateRootHash': None, 'viewNo': 3} since ppSeqNo 21116 is not between 0 and 300)
      

      Workaround:
      Restart whole pool via CLI

      Logs are available on LogProcessor

        Attachments

          Activity

            People

            • Assignee:
              spivachuk Nikita Spivachuk
              Reporter:
              zhigunenko.dsr NIkita Zhigunenko
              Watchers:
              Nikita Spivachuk, NIkita Zhigunenko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: