Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1033

Problems with nodes demotion during load test

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Link:
    • Sprint:
      INDY 18.01: Stability+, Sprint 18.02 Stability, Sprint 18.03 Stability, DKMS

      Description

      Case 1:
      Steps to Reproduce:

      1. Setup the pool of 7 nodes.
      2. Write several transactions.
      3. Disconnect one of nodes (not primary) using following command in CLI:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': []}
      4. Send 1000 tnx using load test (based on indy-sdk).
      5. Connect disconnected node back:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': ['VALIDATOR']}
      6. Send 1000 tnx using load test.

      Actual Results:
      Node which was disconnected stopped processed transactions (it wrote 3027 when the rest ones wrote 3046).

      Expected Results:
      Node which was disconnected should work same as another ones.

      There is a problem to collect extended logs for case 1 because of case 2, so there are only logs with info level for this case: case1_logs.7z

      Case 2:
      Steps to Reproduce:
      1. Setup the pool of 4 nodes.
      2. Demote one of nodes.
      3. Try to write 1000 tnx using load test.

      Actual Results:
      Pool is broken after writing 300 tnx. Following messages appear in logs:

      2017-12-14 11:12:08,017 | WARNING | replica.py (821) | dispatchThreePhaseMsg | Node3:0 stashing 3 phase message PREPREPARE{'reqIdr': [['Th7MpTaRZVRYnPiabds81Y', 1513249927978175705]], 'viewNo': 1, 'ledgerId': 1, 'instId': 0, 'digest': '41597a8d40cc9ae0505aaa50fceb3fd84d2d81b48a200af9b19461f732420a28', 'ppSeqNo': 301, 'txnRootHash': 'AVHPJs38TuQ3puSHBRrvCYz51fMUratYLYn9UHgkBnBV', 'ppTime': 1513249928, 'stateRootHash': 'FYhgkCWazT92EMzrzfRRwZHqh66tDiHfZgPmk1j1jKPF', 'blsMultiSig': ['RXUoQnE2XWcbke2ekzRyx2aaWmpVUzByNkDzMmHxVNuamw8MdYFgcsFPQqvmBNM1ck5YazZ16Kk4kHT5SJwmQFKMybZF5WYDhXZ1n6FKheSSccT48onVEW5WupBkHymAzUpWokoiN38miTKNP3VXVtkQz6o1oHPCcaWC9Be1oRJWaB', ['Node4', 'Node2', 'Node3'], [1, '3yg9fiR6inj6S9GTJbEpYmCKyynhw9CjBFAs4QzVk5Kb', 'GYrER3ijU71jQeiy1uW8ftfagXatTwPps9wMUVa4ZQyt', '8iA1MpeX34en2UVmMtESoMDEJZ2MdQHN7A7enh2yfdDm', 1513249927]], 'discarded': 1} since ppSeqNo 301 is not between 0 and 300

      Expected Results:
      Pool should work.

      Extended logs for Case 2: case2_extended_part1.7zcase2_extended_part2.7zcase2_extended_part3.7z

      Logs for Case 2: Node1.log Node2.logNode3.log Node4.log

        Attachments

        1. case1_logs.7z
          1.03 MB
        2. case2_extended_part1.7z
          7.57 MB
        3. case2_extended_part2.7z
          7.36 MB
        4. case2_extended_part3.7z
          7.42 MB
        5. logs.tar.gz
          4.04 MB
        6. Node1.log
          309 kB
        7. Node2.log
          267 kB
        8. Node3.log
          268 kB
        9. Node4.log
          268 kB

          Issue Links

            Activity

              People

              • Assignee:
                VladimirWork Vladimir Shishkin
                Reporter:
                ozheregelya Olga Zheregelya
                Watchers:
                Dmitry Surnin, Olga Zheregelya, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: