Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-960

Node is broken after load_test.py run

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Link:
    • Sprint:
      INDY 17.24: Node Perf, INDY 17.25

      Description

      Build Info:
      indy-node 1.2.49

      Steps to Reproduce:
      1. Install pool of 4 nodes.
      2. Place load test files to /home/indy at any node (e.g. Node 1).
      3. Run `python3 add_keys.py Steward1 000000000000000000000000Steward1`.
      4. Run `python3 load_test.py -c 10 -r 1000` as indy user (works).
      5. Run `python3 load_test.py -c 20 -r 2000` as indy user (doesn't work).

      Actual Results:
      It's unable to run Step 4 because script can't connect to node where we run load scripts (raise Exception("Not connected fully"). Restarting of this node doesn't help. There is a message in node log about "found node state empty .. try to recreate ledger".

      `read_ledger --type=domain --count` returns the next:

      ERROR:stp_core.common.log:/var/lib/indy/sandbox/data/nodes/Node1-read-copy/domain_merkleNodes does not have position 9842
      Traceback (most recent call last):
      File "/usr/local/bin/read_ledger", line 164, in <module>
      ledger = get_ledger(args.type, read_copy_ledger_data_dir)
      File "/usr/local/bin/read_ledger", line 98, in get_ledger
      return Ledger(CompactMerkleTree(hashStore=hash_store), dataDir=ledger_data_dir, fileName=ledger_name)
      File "/usr/local/lib/python3.5/dist-packages/plenum/common/ledger.py", line 14, in _init_
      super()._init_(*args, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/ledger/ledger.py", line 55, in _init_
      self.recoverTree()
      File "/usr/local/lib/python3.5/dist-packages/ledger/ledger.py", line 73, in recoverTree
      self.recoverTreeFromHashStore()
      File "/usr/local/lib/python3.5/dist-packages/ledger/ledger.py", line 99, in recoverTreeFromHashStore
      treeSize + 1)))
      File "/usr/local/lib/python3.5/dist-packages/ledger/compact_merkle_tree.py", line 220, in inclusion_proof
      for a, b in self._path(start, 0, end)]
      File "/usr/local/lib/python3.5/dist-packages/ledger/compact_merkle_tree.py", line 220, in <listcomp>
      for a, b in self._path(start, 0, end)]
      File "/usr/local/lib/python3.5/dist-packages/ledger/compact_merkle_tree.py", line 211, in merkle_tree_hash
      foldedHash = self.__hasher._hash_fold(hashes[::-1])
      File "/usr/local/lib/python3.5/dist-packages/ledger/tree_hasher.py", line 77, in _hash_fold
      accum = self.hash_children(cur, accum)
      File "/usr/local/lib/python3.5/dist-packages/ledger/tree_hasher.py", line 29, in hash_children
      hasher.update(b"\x01" + left + right)
      TypeError: can't concat bytes to NoneType

        Attachments

        1. Node1_journalctl.txt
          23 kB
        2. Node1.log
          1.34 MB
        3. Node2.log
          5.56 MB

          Issue Links

            Activity

              People

              • Assignee:
                VladimirWork Vladimir Shishkin
                Reporter:
                VladimirWork Vladimir Shishkin
                Watchers:
                Nikita Spivachuk, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: