Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-986

Pool stops working: Node services stop with 1,000 simultaneous clients doing GET_NYM reads

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Build: 1.1.43 of indy node

      Testing Setup
      Global pool setup through AWS across 7 regions.
      5 Globally dispersed client machines each running Libindy and load testing scripts.

      Each client machine using Libindy ran 200 threads each thread sending 2,000 GET_NYM transactions each one at a time. So with 5 Libindy machines it was simulating 1,000 clients each sending 2,000 GET_NYM transactions for a total of 2,000,000 read transactions.

      I successfully ran this with each machine simulating 150 threads per libindy machine for a total of 750 clients across 5 machines doing 1,500,000 GET_NYM transactions.

      After a little over 30,000 reads from the pool the pool stopped working. I started the CLI and tried to connect, but none of the nodes would connect. The logs showed no errors so I checked the status of the node service.

      Here is the error:

      Traceback (most recent call last):
        File "/usr/local/bin/start_sovrin_node", line 19, in <module>
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 289, in __exit__
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 285, in shutdownSync
        File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 275, in shutdown
        File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 259, in run
        File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 222, in runForever
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 205, in runOnceNicely
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 150, in prodAllOnce
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/server/node.py", line 361, in prod
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 742, in prod
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 67, in _serviceActions
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 74, in wrapper
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/validator_info_tool.py", line 171, in dump_json_file
      OSError: [Errno 24] Too many open files: '/home/sovrin/.sovrin/node1_info.json'
      Error in sys.excepthook:
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
        File "<frozen importlib._bootstrap>", line 969, in _find_and_load
        File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 661, in exec_module
        File "<frozen importlib._bootstrap_external>", line 766, in get_code
        File "<frozen importlib._bootstrap_external>", line 818, in get_data
      OSError: [Errno 24] Too many open files: '/usr/lib/python3/dist-packages/apport/__init__.py'
      Original exception was:
      Traceback (most recent call last):
        File "/usr/local/bin/start_sovrin_node", line 19, in <module>
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 289, in __exit__
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 285, in shutdownSync
        File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 275, in shutdown
        File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 259, in run
        File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
        File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 222, in runForever
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 205, in runOnceNicely
        File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 150, in prodAllOnce
        File "/usr/local/lib/python3.5/dist-packages/sovrin_node/server/node.py", line 361, in prod
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 742, in prod
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 67, in _serviceActions
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 74, in wrapper
        File "/usr/local/lib/python3.5/dist-packages/plenum/server/validator_info_tool.py", line 171, in dump_json_file
      OSError: [Errno 24] Too many open files: '/home/sovrin/.sovrin/node1_info.json'
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                VladimirWork Vladimir Shishkin
                Reporter:
                krw910 Kelly Wilson
                Watchers:
                Alexander Shcherbakov, Andrey Kononykhin, Kelly Wilson, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: