Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1033

Problems with nodes demotion during load test

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • Medium
    • Resolution: Done
    • None
    • None
    • None
    • None
    • INDY 18.01: Stability+, Sprint 18.02 Stability, Sprint 18.03 Stability, DKMS

    Description

      Case 1:
      Steps to Reproduce:

      1. Setup the pool of 7 nodes.
      2. Write several transactions.
      3. Disconnect one of nodes (not primary) using following command in CLI:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': []}
      4. Send 1000 tnx using load test (based on indy-sdk).
      5. Connect disconnected node back:
      send NODE dest=Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv data={'alias': 'Node1', 'services': ['VALIDATOR']}
      6. Send 1000 tnx using load test.

      Actual Results:
      Node which was disconnected stopped processed transactions (it wrote 3027 when the rest ones wrote 3046).

      Expected Results:
      Node which was disconnected should work same as another ones.

      There is a problem to collect extended logs for case 1 because of case 2, so there are only logs with info level for this case: case1_logs.7z

      Case 2:
      Steps to Reproduce:
      1. Setup the pool of 4 nodes.
      2. Demote one of nodes.
      3. Try to write 1000 tnx using load test.

      Actual Results:
      Pool is broken after writing 300 tnx. Following messages appear in logs:

      2017-12-14 11:12:08,017 | WARNING | replica.py (821) | dispatchThreePhaseMsg | Node3:0 stashing 3 phase message PREPREPARE{'reqIdr': [['Th7MpTaRZVRYnPiabds81Y', 1513249927978175705]], 'viewNo': 1, 'ledgerId': 1, 'instId': 0, 'digest': '41597a8d40cc9ae0505aaa50fceb3fd84d2d81b48a200af9b19461f732420a28', 'ppSeqNo': 301, 'txnRootHash': 'AVHPJs38TuQ3puSHBRrvCYz51fMUratYLYn9UHgkBnBV', 'ppTime': 1513249928, 'stateRootHash': 'FYhgkCWazT92EMzrzfRRwZHqh66tDiHfZgPmk1j1jKPF', 'blsMultiSig': ['RXUoQnE2XWcbke2ekzRyx2aaWmpVUzByNkDzMmHxVNuamw8MdYFgcsFPQqvmBNM1ck5YazZ16Kk4kHT5SJwmQFKMybZF5WYDhXZ1n6FKheSSccT48onVEW5WupBkHymAzUpWokoiN38miTKNP3VXVtkQz6o1oHPCcaWC9Be1oRJWaB', ['Node4', 'Node2', 'Node3'], [1, '3yg9fiR6inj6S9GTJbEpYmCKyynhw9CjBFAs4QzVk5Kb', 'GYrER3ijU71jQeiy1uW8ftfagXatTwPps9wMUVa4ZQyt', '8iA1MpeX34en2UVmMtESoMDEJZ2MdQHN7A7enh2yfdDm', 1513249927]], 'discarded': 1} since ppSeqNo 301 is not between 0 and 300

      Expected Results:
      Pool should work.

      Extended logs for Case 2: case2_extended_part1.7zcase2_extended_part2.7zcase2_extended_part3.7z

      Logs for Case 2: Node1.log Node2.logNode3.log Node4.log

      Attachments

        1. case1_logs.7z
          1.03 MB
          Olga Zheregelya
        2. case2_extended_part1.7z
          7.57 MB
          Olga Zheregelya
        3. case2_extended_part2.7z
          7.36 MB
          Olga Zheregelya
        4. case2_extended_part3.7z
          7.42 MB
          Olga Zheregelya
        5. logs.tar.gz
          4.04 MB
          Vladimir Shishkin
        6. Node1.log
          309 kB
          Olga Zheregelya
        7. Node2.log
          267 kB
          Olga Zheregelya
        8. Node3.log
          268 kB
          Olga Zheregelya
        9. Node4.log
          268 kB
          Olga Zheregelya

        Issue Links

          Activity

            People

              VladimirWork Vladimir Shishkin
              ozheregelya Olga Zheregelya
              Dmitry Surnin, Olga Zheregelya, Vladimir Shishkin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: