Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1033

Problems with nodes demotion during load test

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Link:
    • Sprint:
      INDY 18.01: Stability+, Sprint 18.02 Stability, Sprint 18.03 Stability, DKMS

      Description

      Case 1:
      Steps to Reproduce:

      1. Setup the pool of 7 nodes.
      2. Write several transactions.
      3. Disconnect one of nodes (not primary) using following command in CLI:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': []}
      4. Send 1000 tnx using load test (based on indy-sdk).
      5. Connect disconnected node back:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': ['VALIDATOR']}
      6. Send 1000 tnx using load test.

      Actual Results:
      Node which was disconnected stopped processed transactions (it wrote 3027 when the rest ones wrote 3046).

      Expected Results:
      Node which was disconnected should work same as another ones.

      There is a problem to collect extended logs for case 1 because of case 2, so there are only logs with info level for this case: case1_logs.7z

      Case 2:
      Steps to Reproduce:
      1. Setup the pool of 4 nodes.
      2. Demote one of nodes.
      3. Try to write 1000 tnx using load test.

      Actual Results:
      Pool is broken after writing 300 tnx. Following messages appear in logs:

      2017-12-14 11:12:08,017 | WARNING | replica.py (821) | dispatchThreePhaseMsg | Node3:0 stashing 3 phase message PREPREPARE{'reqIdr': [['Th7MpTaRZVRYnPiabds81Y', 1513249927978175705]], 'viewNo': 1, 'ledgerId': 1, 'instId': 0, 'digest': '41597a8d40cc9ae0505aaa50fceb3fd84d2d81b48a200af9b19461f732420a28', 'ppSeqNo': 301, 'txnRootHash': 'AVHPJs38TuQ3puSHBRrvCYz51fMUratYLYn9UHgkBnBV', 'ppTime': 1513249928, 'stateRootHash': 'FYhgkCWazT92EMzrzfRRwZHqh66tDiHfZgPmk1j1jKPF', 'blsMultiSig': ['RXUoQnE2XWcbke2ekzRyx2aaWmpVUzByNkDzMmHxVNuamw8MdYFgcsFPQqvmBNM1ck5YazZ16Kk4kHT5SJwmQFKMybZF5WYDhXZ1n6FKheSSccT48onVEW5WupBkHymAzUpWokoiN38miTKNP3VXVtkQz6o1oHPCcaWC9Be1oRJWaB', ['Node4', 'Node2', 'Node3'], [1, '3yg9fiR6inj6S9GTJbEpYmCKyynhw9CjBFAs4QzVk5Kb', 'GYrER3ijU71jQeiy1uW8ftfagXatTwPps9wMUVa4ZQyt', '8iA1MpeX34en2UVmMtESoMDEJZ2MdQHN7A7enh2yfdDm', 1513249927]], 'discarded': 1} since ppSeqNo 301 is not between 0 and 300

      Expected Results:
      Pool should work.

      Extended logs for Case 2: case2_extended_part1.7zcase2_extended_part2.7zcase2_extended_part3.7z

      Logs for Case 2: Node1.log Node2.logNode3.log Node4.log

        Attachments

        1. case1_logs.7z
          1.03 MB
          Olga Zheregelya
        2. case2_extended_part1.7z
          7.57 MB
          Olga Zheregelya
        3. case2_extended_part2.7z
          7.36 MB
          Olga Zheregelya
        4. case2_extended_part3.7z
          7.42 MB
          Olga Zheregelya
        5. logs.tar.gz
          4.04 MB
          Vladimir Shishkin
        6. Node1.log
          309 kB
          Olga Zheregelya
        7. Node2.log
          267 kB
          Olga Zheregelya
        8. Node3.log
          268 kB
          Olga Zheregelya
        9. Node4.log
          268 kB
          Olga Zheregelya

          Issue Links

            Activity

              People

              Assignee:
              VladimirWork Vladimir Shishkin
              Reporter:
              ozheregelya Olga Zheregelya
              Watchers:
              Dmitry Surnin, Olga Zheregelya, Vladimir Shishkin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: