Details
-
Bug
-
Status: Complete
-
High
-
Resolution: Done
-
None
-
None
-
None
-
None
-
INDY 17.25
Description
Build: 1.1.43 of indy node
Testing Setup
Global pool setup through AWS across 7 regions.
5 Globally dispersed client machines each running Libindy and load testing scripts.
Each client machine using Libindy ran 200 threads each thread sending 2,000 GET_NYM transactions each one at a time. So with 5 Libindy machines it was simulating 1,000 clients each sending 2,000 GET_NYM transactions for a total of 2,000,000 read transactions.
I successfully ran this with each machine simulating 150 threads per libindy machine for a total of 750 clients across 5 machines doing 1,500,000 GET_NYM transactions.
After a little over 30,000 reads from the pool the pool stopped working. I started the CLI and tried to connect, but none of the nodes would connect. The logs showed no errors so I checked the status of the node service.
Here is the error:
Traceback (most recent call last): File "/usr/local/bin/start_sovrin_node", line 19, in <module> File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 289, in __exit__ File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 285, in shutdownSync File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 275, in shutdown File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__ File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 259, in run File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 222, in runForever File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 205, in runOnceNicely File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 150, in prodAllOnce File "/usr/local/lib/python3.5/dist-packages/sovrin_node/server/node.py", line 361, in prod File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 742, in prod File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 67, in _serviceActions File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 74, in wrapper File "/usr/local/lib/python3.5/dist-packages/plenum/server/validator_info_tool.py", line 171, in dump_json_file OSError: [Errno 24] Too many open files: '/home/sovrin/.sovrin/node1_info.json' Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook File "<frozen importlib._bootstrap>", line 969, in _find_and_load File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 673, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 661, in exec_module File "<frozen importlib._bootstrap_external>", line 766, in get_code File "<frozen importlib._bootstrap_external>", line 818, in get_data OSError: [Errno 24] Too many open files: '/usr/lib/python3/dist-packages/apport/__init__.py' Original exception was: Traceback (most recent call last): File "/usr/local/bin/start_sovrin_node", line 19, in <module> File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 289, in __exit__ File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 285, in shutdownSync File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 275, in shutdown File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__ File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/local/lib/python3.5/dist-packages/sovrin_node/utils/node_runner.py", line 29, in run_node File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 259, in run File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 222, in runForever File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 205, in runOnceNicely File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 150, in prodAllOnce File "/usr/local/lib/python3.5/dist-packages/sovrin_node/server/node.py", line 361, in prod File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 742, in prod File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 67, in _serviceActions File "/usr/local/lib/python3.5/dist-packages/plenum/server/has_action_queue.py", line 74, in wrapper File "/usr/local/lib/python3.5/dist-packages/plenum/server/validator_info_tool.py", line 171, in dump_json_file OSError: [Errno 24] Too many open files: '/home/sovrin/.sovrin/node1_info.json'
