Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1077

The huge amount of calls and a lot of execution time in kv_store.py

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Build Info:
      indy-node 1.2.255

      Steps to Reproduce:
      1. Install node and stop it.
      2. Place file with 100k json NYMs to node named `txns`.
      3. Run `python3 -m cProfile -s time add_json_txns_to_ledger.py txns`.
      4. Check the cprofile results.

      Actual Results:
      ~5400 seconds from ~6500 are spent in kv_store.py at lines 64 and 69.
      99% of all functions' calls are performed in kv_store.py at line 69.

      10091189092 function calls (10086844765 primitive calls) in 6563.480 seconds
      
      Ordered by: internal time
      
         ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      10003200163 3039.453    0.000 3039.453    0.000 kv_store.py:69(<genexpr>)
         200016 2371.693    0.012 6481.807    0.032 kv_store.py:64(size)
         200026 1064.881    0.005 4104.334    0.021 {built-in method builtins.sum}
         915129   17.407    0.000   25.512    0.000 base58.py:30(b58encode)
         299994    9.021    0.000   10.000    0.000 {method 'Put' of 'leveldb.LevelDB' objects}
       40228222    7.379    0.000    7.379    0.000 {built-in method builtins.divmod}
         400037    4.235    0.000    4.235    0.000 {method 'RangeIter' of 'leveldb.LevelDB' objects}
      1600164/1600034    2.716    0.000    6.400    0.000 typing.py:1043(__subclasscheck__)
         200000    2.160    0.000    8.971    0.000 msgpack_serializer.py:45(__sort_dict)
      7832836/4832307    1.993    0.000    8.831    0.000 {built-in method builtins.isinstance}
      1600227/1600059    1.711    0.000    3.060    0.000 abc.py:194(__subclasscheck__)
         100000    1.695    0.000 6501.976    0.065 compact_merkle_tree.py:95(_push_subtree)
         100000    1.669    0.000 6488.565    0.065 leveldb_hash_store.py:21(writeLeaf)
         400037    1.545    0.000    5.780    0.000 kv_store_leveldb.py:33(iterator)
      1330300/99998    1.463    0.000    3.362    0.000 hash_store.py:87(getNodePosition)
      199994/100000    1.416    0.000    4.084    0.000 compact_merkle_tree.py:139(__push_subtree_hash)
              1    1.386    1.386 6563.483 6563.483 add_json_txns_to_ledger.py:3(<module>)
         100000    1.351    0.000   31.568    0.000 ledger.py:138(_build_merkle_proof)
        3000559    1.347    0.000    1.347    0.000 _weakrefset.py:70(__contains__)
        1630306    1.331    0.000    1.331    0.000 util.py:24(highest_bit_set)
         100000    1.309    0.000    1.309    0.000 decoder.py:345(raw_decode)
         200000    1.134    0.000    2.177    0.000 __init__.py:41(packb)
         299994    1.103    0.000   11.728    0.000 kv_store_leveldb.py:45(put)
         815122    1.075    0.000    2.752    0.000 tree_hasher.py:27(hash_children)
         200000    1.043    0.000    1.043    0.000 {method 'pack' of 'msgpack._packer.Packer' objects}
        1630298    0.992    0.000    0.992    0.000 util.py:5(count_bits_set)
           3415    0.990    0.000    2.251    0.001 inspect.py:690(getmodule)
        1788618    0.978    0.000    0.978    0.000 kv_store_leveldb_int_keys.py:13(compare)
         199998    0.944    0.000    1.734    0.000 compact_merkle_tree.py:35(_update)
        1600000    0.863    0.000    7.262    0.000 typing.py:1035(__instancecheck__)
         200032    0.814    0.000    0.849    0.000 {built-in method builtins.sorted}
         200012    0.746    0.000 6482.553    0.032 leveldb_hash_store.py:59(leafCount)
         915122    0.724    0.000    0.724    0.000 {method 'update' of '_hashlib.HASH' objects}
         915123    0.690    0.000    0.690    0.000 {built-in method _hashlib.openssl_sha256}
         100000    0.688    0.000 6502.664    0.065 compact_merkle_tree.py:156(append)
         915122    0.685    0.000    0.685    0.000 {method 'digest' of '_hashlib.HASH' objects}
         915114    0.679    0.000   26.190    0.000 ledger.py:228(hashToStr)
         100004    0.649    0.000    2.714    0.000 tree_hasher.py:73(_hash_fold)
         200000    0.637    0.000   14.049    0.000 msgpack_serializer.py:21(serialize)
         200979    0.595    0.000    0.595    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
         100000    0.530    0.000 6534.762    0.065 ledger.py:133(_addToTreeSerialized)
         100000    0.521    0.000   22.225    0.000 ledger.py:142(<listcomp>)
      

      Expected Results:
      Investigation of actual results is needed for some optimization fixes.

        Attachments

          Activity

            People

            • Assignee:
              VladimirWork Vladimir Shishkin
              Reporter:
              VladimirWork Vladimir Shishkin
              Watchers:
              Dmitry Surnin, Sergey Minaev, Vladimir Shishkin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: