Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1455

Pool cannot order transactions because Node set incorrect watermarks after it's restart

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • High
    • Resolution: Done
    • None
    • 1.5
    • None
    • None
    • indy-node 1.4.490

    • EV 18.14 Monitoring/Stability

    Description

      Step to reproduce:
      1) setup pool with 25 nodes on AWS
      2) set 1,06kk txns to ledger
      3) run concurrent load test:

      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 286 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 57 -n 1 -k "{\"nym\": {\"count\": 4},\"nym\": {\"attrib\": 1},\"nym\": {\"schema\": 1},\"nym\": {\"cred_def\": 1}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_schema\": {\"file_name\": \"./load_schema/successful\"}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_attrib\": {\"file_name\": \"./load_attrib/successful\"}}"
      
      python3 perf_processes.py -g pool_transactions_genesis -b 10 -r 2 -m t -c 143 -t 4 -n 1 -k "{\"get_nym\": {\"file_name\": \"./load_nym/successful\"}}"
      

      Actual results:

      2018-07-05 07:16:58,838 | INFO     | replica.py           ( 884) | dispatchThreePhaseMsg | REPLICA:(Node18:5) Node18:5 stashing 3 phase message PREPREPARE{'discarded': 1, 'digest': '28eb27c43db6b7472b72e9fa547f7fc7e7f1eb8e6d4fc521c225a98c5fc3ca91', 'reqIdr': ['2f051e9054ea521e0acce5949839e3a3f174ba6bc063974b6fb796f04564dc34'], 'ppSeqNo': 21116, 'instId': 5, 'ppTime': 1530775018, 'ledgerId': 1, 'txnRootHash': None, 'stateRootHash': None, 'viewNo': 3} since ppSeqNo 21116 is not between 0 and 300)
      

      Workaround:
      Restart whole pool via CLI

      Logs are available on LogProcessor

      Attachments

        Activity

          People

            spivachuk Nikita Spivachuk
            zhigunenko.dsr NIkita Zhigunenko
            Nikita Spivachuk, NIkita Zhigunenko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: