2017-08-28 15:16:42,338 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node6.log 2017-08-28 15:16:42,339 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-08-28 15:16:56,551 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-08-28 15:16:57,078 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-08-28 15:16:57,780 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-08-28 15:16:57,814 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-08-28 15:17:05,808 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-08-28 15:17:05,823 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-08-28 15:17:05,905 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:17:05,933 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:17:05,996 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.06267848197603598 seconds 2017-08-28 15:17:06,087 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node6 2017-08-28 15:17:06,110 | INFO | node.py (2378) | initStateFromLedger | Node6 found state to be empty, recreating from ledger 2017-08-28 15:17:06,217 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:17:06,249 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:17:06,305 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.05615487799514085 seconds 2017-08-28 15:17:06,324 | INFO | node.py (2378) | initStateFromLedger | Node6 found state to be empty, recreating from ledger 2017-08-28 15:17:06,775 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-08-28 15:17:06,777 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:17:06,777 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:17:06,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-08-28 15:17:06,791 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:17:06,800 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:17:06,801 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-08-28 15:17:06,803 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-08-28 15:17:06,807 | DEBUG | replica.py ( 311) | h | Node6:0 set watermarks as 0 300 2017-08-28 15:17:06,808 | DISPLAY | replicas.py ( 36) | grow | Node6 added replica Node6:0 to instance 0 (master) 2017-08-28 15:17:06,808 | DEBUG | replica.py ( 311) | h | Node6:1 set watermarks as 0 300 2017-08-28 15:17:06,808 | DISPLAY | replicas.py ( 36) | grow | Node6 added replica Node6:1 to instance 1 (backup) 2017-08-28 15:17:06,809 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-08-28 15:17:06,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-08-28 15:17:06,824 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-08-28 15:17:06,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-08-28 15:17:06,825 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-08-28 15:17:06,825 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-08-28 15:17:06,825 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-08-28 15:17:06,825 | DEBUG | node.py ( 337) | __init__ | total plugins loaded in node: 0 2017-08-28 15:17:06,914 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:17:06,943 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:17:06,999 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.05602344899671152 seconds 2017-08-28 15:17:07,027 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node6 has no upgrade events 2017-08-28 15:17:07,028 | INFO | node.py (2378) | initStateFromLedger | Node6 found state to be empty, recreating from ledger 2017-08-28 15:17:07,028 | DEBUG | motor.py ( 34) | set_status | Node6 changing status from stopped to starting 2017-08-28 15:17:07,028 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:17:07,031 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:17:07,031 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:17:07,035 | DEBUG | zstack.py ( 319) | start | Node6 starting with restricted as True and reSetupAuth as True 2017-08-28 15:17:07,036 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-08-28 15:17:07,036 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:17:07,039 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node6/public_keys] 2017-08-28 15:17:07,047 | DEBUG | zstack.py ( 347) | open | Node6 will bind its listener at 9701 2017-08-28 15:17:07,048 | INFO | stacks.py ( 84) | start | CONNECTION: Node6 listening for other nodes at 0.0.0.0:9701 2017-08-28 15:17:07,048 | DEBUG | zstack.py ( 319) | start | Node6C starting with restricted as False and reSetupAuth as True 2017-08-28 15:17:07,048 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-08-28 15:17:07,067 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:17:07,067 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-08-28 15:17:07,080 | DEBUG | zstack.py ( 347) | open | Node6C will bind its listener at 9702 2017-08-28 15:17:07,081 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-08-28 15:17:07,081 | INFO | node.py ( 612) | start | Node6 first time running... 2017-08-28 15:17:07,174 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node6 found the following missing connections: Node1, Node3, Node2, Node4 2017-08-28 15:17:07,180 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408624 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:07,181 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node1 at 10.0.0.2:9701 2017-08-28 15:17:07,202 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:17:07,242 | TRACE | remote.py ( 84) | connect | connecting socket 81 45414992 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:07,242 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node3 at 10.0.0.4:9705 2017-08-28 15:17:07,243 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:17:07,250 | TRACE | remote.py ( 84) | connect | connecting socket 83 45419616 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:07,260 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node2 at 10.0.0.3:9703 2017-08-28 15:17:07,260 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:17:07,280 | TRACE | remote.py ( 84) | connect | connecting socket 85 45599872 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:07,280 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node4 at 10.0.0.5:9707 2017-08-28 15:17:07,281 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:17:07,294 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:17:07,300 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.204 seconds 2017-08-28 15:17:07,355 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:17:07,355 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:17:07,356 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:17:07,360 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:17:07,361 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:17:07,362 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:17:07,362 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:17:07,377 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:17:16,854 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 1 2017-08-28 15:17:16,855 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:17:16,863 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-08-28 15:17:22,175 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:22,181 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:22,182 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:22,182 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:22,182 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:17:22,182 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:17:22,185 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:17:22,185 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:17:22,185 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:17:22,267 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:17:22,272 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:17:22,273 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:17:22,276 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:17:22,279 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:17:22,288 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:17:22,288 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:17:22,295 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:17:26,869 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 4 2017-08-28 15:17:26,869 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:17:26,876 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-08-28 15:17:36,890 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 5 2017-08-28 15:17:36,911 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:17:36,912 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-08-28 15:17:37,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:37,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:37,179 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:37,180 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:37,183 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:37,187 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:37,188 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408624 2017-08-28 15:17:37,191 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:37,192 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:17:37,193 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:37,236 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:37,237 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45414992 2017-08-28 15:17:37,237 | TRACE | remote.py ( 84) | connect | connecting socket 87 45657040 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:37,238 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:17:37,238 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:37,238 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:37,263 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 83 45419616 2017-08-28 15:17:37,264 | TRACE | remote.py ( 84) | connect | connecting socket 88 45692640 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:37,265 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:17:37,265 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:37,266 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:37,270 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45599872 2017-08-28 15:17:37,270 | TRACE | remote.py ( 84) | connect | connecting socket 89 45728480 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:37,270 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:17:37,271 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:17:37,288 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.125 seconds 2017-08-28 15:17:37,350 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:17:37,351 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:17:37,351 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:17:37,351 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:17:37,363 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:17:37,364 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:17:37,364 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:17:37,372 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:17:46,938 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 6 2017-08-28 15:17:46,938 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:17:46,939 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-08-28 15:17:52,198 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:52,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:52,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:52,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:52,209 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:17:52,216 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:17:52,217 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:17:52,224 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:17:52,224 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:17:52,255 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:17:52,256 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:17:52,256 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:17:52,266 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:17:52,267 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:17:52,267 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:17:52,267 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:17:52,268 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:17:56,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 7 2017-08-28 15:17:56,962 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:17:56,963 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-08-28 15:18:06,835 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 2 2017-08-28 15:18:06,835 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:18:06,836 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 9 to run in 60 seconds 2017-08-28 15:18:06,836 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 1 2017-08-28 15:18:06,836 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:18:06,837 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-08-28 15:18:06,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 8 2017-08-28 15:18:06,978 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:06,979 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-08-28 15:18:07,098 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action propose_view_change with id 3 2017-08-28 15:18:07,108 | DEBUG | throttler.py ( 31) | acquire | now: 249783.257324465, len(actionsLog): 0 2017-08-28 15:18:07,109 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:18:07,110 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:18:07,110 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-08-28 15:18:07,126 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-08-28 15:18:07,135 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} to all recipients: ['Node1', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:18:07,143 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 1 2017-08-28 15:18:07,144 | INFO | node.py (2089) | propose_view_change | Node6 sent view change since was disconnected from primary for too long 2017-08-28 15:18:07,144 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 1 2017-08-28 15:18:07,146 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:18:07,146 | DEBUG | zstack.py ( 726) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:18:07,149 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:18:07,150 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:18:07,150 | DEBUG | zstack.py ( 726) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:18:07,150 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:18:07,150 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:18:07,150 | DEBUG | zstack.py ( 726) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:18:07,162 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:18:07,163 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:18:07,163 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:18:07,163 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":1,"reason":26,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:18:07,197 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.101 seconds 2017-08-28 15:18:07,232 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:07,232 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:07,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:07,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:07,236 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:07,239 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:07,239 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:18:07,266 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:07,266 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:18:07,266 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:07,268 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:07,268 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45657040 2017-08-28 15:18:07,269 | TRACE | remote.py ( 84) | connect | connecting socket 84 45535904 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:07,271 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:18:07,272 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:07,272 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:07,278 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45692640 2017-08-28 15:18:07,283 | TRACE | remote.py ( 84) | connect | connecting socket 85 45571504 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:07,290 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:18:07,290 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:07,310 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:07,314 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 89 45728480 2017-08-28 15:18:07,315 | TRACE | remote.py ( 84) | connect | connecting socket 83 45641136 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:07,327 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:18:07,328 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:18:07,329 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.119 seconds 2017-08-28 15:18:07,354 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:18:07,406 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:18:07,407 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:18:07,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:18:07,412 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:18:07,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:18:07,413 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:18:07,423 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:18:16,993 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 10 2017-08-28 15:18:17,010 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:17,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-08-28 15:18:22,235 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:22,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:22,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:22,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:22,237 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:18:22,237 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:18:22,238 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:18:22,238 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:18:22,244 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:18:22,276 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:18:22,280 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:18:22,280 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:18:22,281 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:18:22,281 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:18:22,281 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:18:22,282 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:18:22,283 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:18:27,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 11 2017-08-28 15:18:27,025 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:27,025 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-08-28 15:18:37,031 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 12 2017-08-28 15:18:37,038 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:37,038 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-08-28 15:18:37,246 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:37,267 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:37,268 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:37,268 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:37,268 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:37,269 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:37,269 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:18:37,272 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:37,283 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:18:37,301 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:37,302 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:37,302 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 84 45535904 2017-08-28 15:18:37,308 | TRACE | remote.py ( 84) | connect | connecting socket 81 45629936 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:37,324 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:18:37,324 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:37,353 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:37,353 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45571504 2017-08-28 15:18:37,354 | TRACE | remote.py ( 84) | connect | connecting socket 87 45568736 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:37,354 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:18:37,373 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:37,374 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:37,389 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 83 45641136 2017-08-28 15:18:37,392 | TRACE | remote.py ( 84) | connect | connecting socket 88 45631584 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:37,407 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:18:37,415 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:18:37,423 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.182 seconds 2017-08-28 15:18:37,480 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:18:37,480 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:18:37,483 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:18:37,494 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:18:37,495 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:18:37,514 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:18:37,515 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:18:37,515 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:18:47,059 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 13 2017-08-28 15:18:47,059 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:47,064 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-08-28 15:18:52,260 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:52,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:52,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:52,272 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:52,272 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:18:52,272 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:18:52,279 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:18:52,282 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:18:52,283 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:18:52,311 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:18:52,319 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:18:52,320 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:18:52,320 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:18:52,327 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:18:52,327 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:18:52,328 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:18:52,328 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:18:57,066 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 14 2017-08-28 15:18:57,077 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:18:57,078 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-08-28 15:19:06,838 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 9 2017-08-28 15:19:06,838 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:19:06,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 16 to run in 60 seconds 2017-08-28 15:19:06,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 2 2017-08-28 15:19:06,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:19:06,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-08-28 15:19:07,089 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 15 2017-08-28 15:19:07,089 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:07,097 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-08-28 15:19:07,270 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:07,279 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:07,280 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:07,280 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:07,281 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:07,281 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:07,282 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:19:07,308 | TRACE | remote.py ( 84) | connect | connecting socket 78 45571456 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:07,309 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:19:07,310 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:07,310 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:07,310 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45629936 2017-08-28 15:19:07,316 | TRACE | remote.py ( 84) | connect | connecting socket 81 45629936 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:07,352 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:19:07,352 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:07,352 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:07,363 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45568736 2017-08-28 15:19:07,364 | TRACE | remote.py ( 84) | connect | connecting socket 85 45408384 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:07,364 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:19:07,364 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:07,365 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:07,365 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45631584 2017-08-28 15:19:07,383 | TRACE | remote.py ( 84) | connect | connecting socket 83 45410032 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:07,384 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:19:07,384 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:19:07,413 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.146 seconds 2017-08-28 15:19:07,435 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:19:07,477 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:19:07,478 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:19:07,478 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:19:07,478 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:19:07,479 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:19:07,480 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:19:07,480 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:19:17,116 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 17 2017-08-28 15:19:17,126 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:17,126 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-08-28 15:19:22,285 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:22,290 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:22,298 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:22,298 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:22,299 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:19:22,303 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:19:22,312 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:19:22,312 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:19:22,312 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:19:22,340 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:19:22,341 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:19:22,341 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:19:22,342 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:19:22,345 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:19:22,347 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:19:22,349 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:19:22,350 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:19:27,130 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 18 2017-08-28 15:19:27,140 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:27,141 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-08-28 15:19:37,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 19 2017-08-28 15:19:37,148 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:37,148 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-08-28 15:19:37,294 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:37,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:37,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:37,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:37,300 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:37,309 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:37,309 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45571456 2017-08-28 15:19:37,311 | TRACE | remote.py ( 84) | connect | connecting socket 78 45632512 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:37,325 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:19:37,325 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:37,325 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:37,330 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45629936 2017-08-28 15:19:37,332 | TRACE | remote.py ( 84) | connect | connecting socket 81 45624640 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:37,333 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:19:37,365 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:37,365 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:37,365 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45408384 2017-08-28 15:19:37,366 | TRACE | remote.py ( 84) | connect | connecting socket 87 45629936 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:37,387 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:19:37,387 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:37,388 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:37,388 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 83 45410032 2017-08-28 15:19:37,391 | TRACE | remote.py ( 84) | connect | connecting socket 88 45571856 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:37,391 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:19:37,392 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:19:37,392 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.113 seconds 2017-08-28 15:19:37,433 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:19:37,474 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:19:37,475 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:19:37,475 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:19:37,481 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:19:37,485 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:19:37,485 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:19:37,486 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:19:47,154 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 20 2017-08-28 15:19:47,167 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:47,167 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-08-28 15:19:52,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:52,314 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:52,314 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:52,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:52,318 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:19:52,320 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:19:52,320 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:19:52,325 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:19:52,326 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:19:52,358 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:19:52,358 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:19:52,359 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:19:52,360 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:19:52,368 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:19:52,372 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:19:52,372 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:19:52,372 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:19:57,182 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 21 2017-08-28 15:19:57,185 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:19:57,188 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-08-28 15:20:06,860 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 16 2017-08-28 15:20:06,867 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:20:06,868 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 23 to run in 60 seconds 2017-08-28 15:20:06,868 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 3 2017-08-28 15:20:06,868 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:20:06,868 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-08-28 15:20:07,214 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 22 2017-08-28 15:20:07,215 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:07,216 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-08-28 15:20:07,323 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:07,328 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:07,328 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:07,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:07,339 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:07,340 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:07,340 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45632512 2017-08-28 15:20:07,344 | TRACE | remote.py ( 84) | connect | connecting socket 78 45632000 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:07,347 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:20:07,350 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:07,351 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:07,352 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45624640 2017-08-28 15:20:07,364 | TRACE | remote.py ( 84) | connect | connecting socket 81 45408384 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:07,365 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:20:07,365 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:07,365 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:07,372 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45629936 2017-08-28 15:20:07,381 | TRACE | remote.py ( 84) | connect | connecting socket 82 45410032 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:07,381 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:20:07,382 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:07,417 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:07,417 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45571856 2017-08-28 15:20:07,418 | TRACE | remote.py ( 84) | connect | connecting socket 86 45629936 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:07,440 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:20:07,441 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:20:07,442 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.131 seconds 2017-08-28 15:20:07,474 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:20:07,490 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:20:07,492 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:20:07,493 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:20:07,499 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:20:07,506 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:20:07,514 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:20:07,515 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:20:17,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 24 2017-08-28 15:20:17,223 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:17,224 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-08-28 15:20:22,331 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:22,334 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:22,339 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:22,340 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:22,340 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:20:22,340 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:20:22,351 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:20:22,352 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:20:22,353 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:20:22,387 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:20:22,387 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:20:22,389 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:20:22,390 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:20:22,390 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:20:22,390 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:20:22,396 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:20:22,396 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:20:27,235 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 25 2017-08-28 15:20:27,241 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:27,241 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-08-28 15:20:37,243 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 26 2017-08-28 15:20:37,248 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:37,254 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-08-28 15:20:37,356 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:37,356 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:37,357 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:37,357 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:37,357 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:37,358 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:37,358 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45632000 2017-08-28 15:20:37,375 | TRACE | remote.py ( 84) | connect | connecting socket 78 45634160 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:37,375 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:20:37,379 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:37,380 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:37,380 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45408384 2017-08-28 15:20:37,383 | TRACE | remote.py ( 84) | connect | connecting socket 81 45408384 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:37,386 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:20:37,386 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:37,394 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:37,394 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 82 45410032 2017-08-28 15:20:37,394 | TRACE | remote.py ( 84) | connect | connecting socket 82 45410032 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:37,395 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:20:37,395 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:37,395 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:37,395 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45629936 2017-08-28 15:20:37,440 | TRACE | remote.py ( 84) | connect | connecting socket 87 45571856 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:37,440 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:20:37,444 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:20:37,444 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.108 seconds 2017-08-28 15:20:37,488 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:20:37,509 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:20:37,509 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:20:37,511 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:20:37,512 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:20:37,512 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:20:37,533 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:20:37,533 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:20:47,283 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 27 2017-08-28 15:20:47,288 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:47,288 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-08-28 15:20:52,364 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:52,364 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:52,374 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:52,375 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:52,375 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:20:52,379 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:20:52,386 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:20:52,387 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:20:52,387 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:20:52,408 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:20:52,409 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:20:52,411 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:20:52,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:20:52,412 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:20:52,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:20:52,413 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:20:52,413 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:20:57,306 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 28 2017-08-28 15:20:57,309 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:20:57,311 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-08-28 15:21:06,878 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 23 2017-08-28 15:21:06,889 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:21:06,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 30 to run in 60 seconds 2017-08-28 15:21:06,892 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 4 2017-08-28 15:21:06,892 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:21:06,892 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-08-28 15:21:07,316 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 29 2017-08-28 15:21:07,319 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:07,334 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-08-28 15:21:07,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:07,371 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:07,371 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:07,385 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:07,386 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:07,389 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:07,391 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45634160 2017-08-28 15:21:07,392 | TRACE | remote.py ( 84) | connect | connecting socket 78 45634032 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:07,393 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:21:07,393 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:07,393 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:07,413 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45408384 2017-08-28 15:21:07,422 | TRACE | remote.py ( 84) | connect | connecting socket 81 45408384 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:07,423 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:21:07,423 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:07,453 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:07,453 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 82 45410032 2017-08-28 15:21:07,454 | TRACE | remote.py ( 84) | connect | connecting socket 86 45629936 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:07,454 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:21:07,454 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:07,455 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:07,455 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45571856 2017-08-28 15:21:07,455 | TRACE | remote.py ( 84) | connect | connecting socket 88 45706288 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:07,456 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:21:07,475 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:21:07,476 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.110 seconds 2017-08-28 15:21:07,535 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:21:07,552 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:21:07,552 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:21:07,552 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:21:07,564 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:21:07,564 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:21:07,566 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:21:07,567 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:21:17,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 31 2017-08-28 15:21:17,352 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:17,357 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-08-28 15:21:22,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:22,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:22,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:22,384 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:22,384 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:21:22,394 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:21:22,395 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:21:22,395 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:21:22,395 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:21:22,431 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:21:22,434 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:21:22,435 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:21:22,442 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:21:22,444 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:21:22,444 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:21:22,445 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:21:22,445 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:21:27,377 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 32 2017-08-28 15:21:27,378 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:27,378 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-08-28 15:21:37,383 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 33 2017-08-28 15:21:37,390 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:37,391 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-08-28 15:21:37,396 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:37,407 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:37,408 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:37,408 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:37,408 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:37,410 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:37,423 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45634032 2017-08-28 15:21:37,427 | TRACE | remote.py ( 84) | connect | connecting socket 78 45410032 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:37,467 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:21:37,468 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:37,468 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:37,475 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45408384 2017-08-28 15:21:37,484 | TRACE | remote.py ( 84) | connect | connecting socket 85 45570736 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:37,484 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:21:37,484 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:37,485 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:37,485 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45629936 2017-08-28 15:21:37,501 | TRACE | remote.py ( 84) | connect | connecting socket 87 45590336 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:37,502 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:21:37,502 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:37,504 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:37,514 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45706288 2017-08-28 15:21:37,542 | TRACE | remote.py ( 84) | connect | connecting socket 84 45408384 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:37,542 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:21:37,542 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:21:37,578 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:21:37,579 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:21:37,583 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:21:37,584 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:21:37,584 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:21:37,584 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:21:37,608 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:21:37,618 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:21:37,628 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.251 seconds 2017-08-28 15:21:47,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 34 2017-08-28 15:21:47,407 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:47,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 35 to run in 10 seconds 2017-08-28 15:21:52,419 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:52,420 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:52,424 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:52,425 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:52,433 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:21:52,433 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:21:52,434 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:21:52,434 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:21:52,443 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:21:52,474 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:21:52,475 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:21:52,484 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:21:52,484 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:21:52,487 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:21:52,495 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:21:52,496 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:21:52,496 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:21:57,422 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 35 2017-08-28 15:21:57,423 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:21:57,430 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-08-28 15:22:06,903 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 30 2017-08-28 15:22:06,911 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:22:06,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 37 to run in 60 seconds 2017-08-28 15:22:06,914 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 5 2017-08-28 15:22:06,922 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:22:06,929 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-08-28 15:22:07,440 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 36 2017-08-28 15:22:07,462 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:07,462 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-08-28 15:22:07,463 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:07,468 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:07,469 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:07,469 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:07,471 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:07,472 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:07,484 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45410032 2017-08-28 15:22:07,484 | TRACE | remote.py ( 84) | connect | connecting socket 78 45410032 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:07,488 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:22:07,488 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:07,488 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:07,489 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45570736 2017-08-28 15:22:07,501 | TRACE | remote.py ( 84) | connect | connecting socket 81 45570736 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:07,502 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:22:07,503 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:07,515 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:07,551 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45590336 2017-08-28 15:22:07,551 | TRACE | remote.py ( 84) | connect | connecting socket 86 45629936 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:07,552 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:22:07,552 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:07,552 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:07,552 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 84 45408384 2017-08-28 15:22:07,571 | TRACE | remote.py ( 84) | connect | connecting socket 88 45632704 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:07,571 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:22:07,596 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:22:07,621 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:22:07,647 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:22:07,649 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:22:07,649 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:22:07,650 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:22:07,651 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:22:07,658 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:22:07,663 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:22:07,664 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.226 seconds 2017-08-28 15:22:17,463 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 38 2017-08-28 15:22:17,468 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:17,468 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-08-28 15:22:22,474 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:22,479 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:22,480 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:22,482 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:22,499 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:22:22,500 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:22:22,500 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:22:22,500 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:22:22,504 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:22:22,536 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:22:22,536 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:22:22,543 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:22:22,551 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:22:22,556 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:22:22,556 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:22:22,559 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:22:22,560 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:22:27,499 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 39 2017-08-28 15:22:27,519 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:27,523 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-08-28 15:22:37,489 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:37,492 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:37,493 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:37,494 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:37,494 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:37,495 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:37,512 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45410032 2017-08-28 15:22:37,514 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:37,523 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:22:37,523 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:37,533 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:37,533 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45570736 2017-08-28 15:22:37,534 | TRACE | remote.py ( 84) | connect | connecting socket 81 45410032 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:37,534 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:22:37,552 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:37,588 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:37,589 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45629936 2017-08-28 15:22:37,590 | TRACE | remote.py ( 84) | connect | connecting socket 85 45658160 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:37,591 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:22:37,591 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:37,619 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:37,620 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45632704 2017-08-28 15:22:37,620 | TRACE | remote.py ( 84) | connect | connecting socket 87 45659808 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:37,621 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:22:37,621 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:22:37,622 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.151 seconds 2017-08-28 15:22:37,658 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 40 2017-08-28 15:22:37,691 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:37,692 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-08-28 15:22:37,692 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:22:37,705 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:22:37,723 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:22:37,723 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:22:37,724 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:22:37,724 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:22:37,724 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:22:37,726 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:22:37,790 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.157 seconds 2017-08-28 15:22:47,720 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 41 2017-08-28 15:22:47,723 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:47,724 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-08-28 15:22:52,493 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:52,500 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:52,500 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:52,500 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:52,501 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:22:52,501 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:22:52,515 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:22:52,515 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:22:52,516 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:22:52,546 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:22:52,546 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:22:52,546 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:22:52,547 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:22:52,547 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:22:52,547 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:22:52,547 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:22:52,551 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:22:57,725 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 42 2017-08-28 15:22:57,727 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:22:57,731 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 43 to run in 10 seconds 2017-08-28 15:23:06,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 37 2017-08-28 15:23:06,931 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:23:06,932 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 44 to run in 60 seconds 2017-08-28 15:23:06,932 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 6 2017-08-28 15:23:06,940 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:23:06,941 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-08-28 15:23:07,504 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:07,508 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:07,508 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:07,508 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:07,509 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:07,524 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:07,524 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:23:07,532 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:07,532 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:23:07,556 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:07,556 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:07,556 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45410032 2017-08-28 15:23:07,583 | TRACE | remote.py ( 84) | connect | connecting socket 81 45410032 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:07,584 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:23:07,610 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:07,611 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:07,623 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45658160 2017-08-28 15:23:07,624 | TRACE | remote.py ( 84) | connect | connecting socket 86 45629936 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:07,624 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:23:07,628 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:07,630 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:07,631 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45659808 2017-08-28 15:23:07,643 | TRACE | remote.py ( 84) | connect | connecting socket 88 45632704 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:07,644 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:23:07,644 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:23:07,672 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.187 seconds 2017-08-28 15:23:07,678 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:23:07,708 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:23:07,710 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:23:07,711 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:23:07,724 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:23:07,724 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:23:07,729 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:23:07,729 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:23:07,750 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 43 2017-08-28 15:23:07,808 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:07,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 45 to run in 10 seconds 2017-08-28 15:23:17,855 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 45 2017-08-28 15:23:17,857 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:17,875 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-08-28 15:23:22,504 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:22,518 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:22,519 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:22,520 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:22,520 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:23:22,521 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:23:22,522 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:23:22,524 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:23:22,542 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:23:22,578 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:23:22,579 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:23:22,580 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:23:22,584 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:23:22,592 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:23:22,598 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:23:22,599 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:23:22,608 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:23:27,896 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 46 2017-08-28 15:23:27,899 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:27,899 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-08-28 15:23:37,506 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:37,507 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:37,511 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:37,511 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:37,512 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:37,512 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:37,513 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:23:37,519 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:37,529 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:23:37,529 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:37,529 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:37,531 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45410032 2017-08-28 15:23:37,546 | TRACE | remote.py ( 84) | connect | connecting socket 81 45410032 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:37,546 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:23:37,575 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:37,576 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:37,576 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45629936 2017-08-28 15:23:37,577 | TRACE | remote.py ( 84) | connect | connecting socket 85 45623680 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:37,577 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:23:37,595 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:37,600 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:37,602 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45632704 2017-08-28 15:23:37,602 | TRACE | remote.py ( 84) | connect | connecting socket 87 45603568 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:37,603 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:23:37,606 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:23:37,606 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.121 seconds 2017-08-28 15:23:37,667 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:23:37,677 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:23:37,677 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:23:37,677 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:23:37,679 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:23:37,679 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:23:37,687 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:23:37,710 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:23:37,912 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 47 2017-08-28 15:23:37,912 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:37,912 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-08-28 15:23:47,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 48 2017-08-28 15:23:47,931 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:47,932 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-08-28 15:23:52,516 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:52,517 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:52,517 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:52,522 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:52,523 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:23:52,523 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:23:52,527 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:23:52,527 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:23:52,528 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:23:52,555 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:23:52,556 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:23:52,556 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:23:52,556 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:23:52,567 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:23:52,568 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:23:52,568 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:23:52,571 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:23:57,948 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 49 2017-08-28 15:23:57,948 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:23:57,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-08-28 15:24:06,943 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 44 2017-08-28 15:24:06,944 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:24:06,959 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 51 to run in 60 seconds 2017-08-28 15:24:06,960 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 7 2017-08-28 15:24:06,960 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:24:06,960 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-08-28 15:24:07,518 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:07,521 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:07,521 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:07,522 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:07,522 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:07,527 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:07,536 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:24:07,537 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:07,541 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:24:07,541 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:07,545 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:07,545 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45410032 2017-08-28 15:24:07,546 | TRACE | remote.py ( 84) | connect | connecting socket 81 45410032 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:07,575 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:24:07,594 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:07,594 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:07,594 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45623680 2017-08-28 15:24:07,612 | TRACE | remote.py ( 84) | connect | connecting socket 86 45631536 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:07,612 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:24:07,614 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:07,614 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:07,615 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45603568 2017-08-28 15:24:07,621 | TRACE | remote.py ( 84) | connect | connecting socket 88 45621632 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:07,622 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:24:07,622 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:24:07,622 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.107 seconds 2017-08-28 15:24:07,686 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:24:07,691 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:24:07,697 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:24:07,697 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:24:07,697 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:24:07,712 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:24:07,713 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:24:07,713 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:24:07,958 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 50 2017-08-28 15:24:07,969 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:07,969 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 52 to run in 10 seconds 2017-08-28 15:24:17,997 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 52 2017-08-28 15:24:18,000 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:18,001 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-08-28 15:24:22,526 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:22,547 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:22,547 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:22,547 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:22,547 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:24:22,548 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:24:22,548 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:24:22,548 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:24:22,548 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:24:22,590 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:24:22,591 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:24:22,595 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:24:22,595 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:24:22,595 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:24:22,596 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:24:22,596 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:24:22,598 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:24:28,027 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 53 2017-08-28 15:24:28,027 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:28,028 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-08-28 15:24:37,533 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:37,533 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:37,534 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:37,543 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:37,544 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:37,544 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:37,544 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:24:37,555 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:37,556 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:24:37,556 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:37,556 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:37,558 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45410032 2017-08-28 15:24:37,561 | TRACE | remote.py ( 84) | connect | connecting socket 81 45410032 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:37,561 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:24:37,563 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:37,597 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:37,598 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45631536 2017-08-28 15:24:37,615 | TRACE | remote.py ( 84) | connect | connecting socket 85 45723824 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:37,616 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:24:37,636 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:37,636 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:37,637 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45621632 2017-08-28 15:24:37,640 | TRACE | remote.py ( 84) | connect | connecting socket 87 45623280 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:37,640 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:24:37,652 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:24:37,654 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.128 seconds 2017-08-28 15:24:37,673 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:24:37,702 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:24:37,702 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:24:37,703 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:24:37,703 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:24:37,713 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:24:37,714 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:24:37,714 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:24:38,048 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 54 2017-08-28 15:24:38,048 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:38,049 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-08-28 15:24:48,058 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 55 2017-08-28 15:24:48,066 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:48,069 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-08-28 15:24:52,540 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:52,544 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:52,544 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:52,544 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:52,544 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:24:52,545 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:24:52,567 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:24:52,567 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:24:52,567 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:24:52,607 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:24:52,607 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:24:52,608 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:24:52,608 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:24:52,612 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:24:52,612 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:24:52,622 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:24:52,622 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:24:58,079 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 56 2017-08-28 15:24:58,083 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:24:58,084 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-08-28 15:25:06,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 51 2017-08-28 15:25:06,975 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:25:06,977 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 58 to run in 60 seconds 2017-08-28 15:25:06,980 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 8 2017-08-28 15:25:06,980 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:25:06,981 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-08-28 15:25:07,547 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:07,547 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:07,548 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:07,548 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:07,548 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:07,550 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:07,550 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:25:07,570 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:07,571 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:25:07,575 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:07,606 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:07,609 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45410032 2017-08-28 15:25:07,616 | TRACE | remote.py ( 84) | connect | connecting socket 86 45631536 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:07,616 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:25:07,642 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:07,642 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:07,642 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45723824 2017-08-28 15:25:07,645 | TRACE | remote.py ( 84) | connect | connecting socket 88 45621376 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:07,655 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:25:07,664 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:07,664 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:07,664 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45623280 2017-08-28 15:25:07,665 | TRACE | remote.py ( 84) | connect | connecting socket 82 45410032 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:07,665 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:25:07,665 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:25:07,666 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.139 seconds 2017-08-28 15:25:07,738 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:25:07,757 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:25:07,757 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:25:07,758 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:25:07,772 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:25:07,776 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:25:07,776 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:25:07,797 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:25:08,108 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 57 2017-08-28 15:25:08,110 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:08,112 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 59 to run in 10 seconds 2017-08-28 15:25:18,127 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 59 2017-08-28 15:25:18,132 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:18,132 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-08-28 15:25:22,549 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:22,562 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:22,563 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:22,563 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:22,563 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:25:22,564 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:25:22,564 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:25:22,567 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:25:22,567 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:25:22,603 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:25:22,604 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:25:22,604 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:25:22,607 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:25:22,607 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:25:22,613 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:25:22,613 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:25:22,614 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:25:28,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 60 2017-08-28 15:25:28,148 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:28,148 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 61 to run in 10 seconds 2017-08-28 15:25:37,560 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:37,571 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:37,571 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:37,572 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:37,572 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:37,572 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:37,583 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:25:37,611 | TRACE | remote.py ( 84) | connect | connecting socket 78 45408384 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:37,611 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:25:37,612 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:37,626 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:37,627 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45631536 2017-08-28 15:25:37,634 | TRACE | remote.py ( 84) | connect | connecting socket 85 45722704 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:37,642 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:25:37,642 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:37,647 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:37,658 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45621376 2017-08-28 15:25:37,661 | TRACE | remote.py ( 84) | connect | connecting socket 87 45623280 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:37,661 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:25:37,666 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:37,666 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:37,672 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 82 45410032 2017-08-28 15:25:37,685 | TRACE | remote.py ( 84) | connect | connecting socket 86 45624928 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:37,698 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:25:37,699 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:25:37,700 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.142 seconds 2017-08-28 15:25:37,736 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:25:37,751 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:25:37,751 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:25:37,751 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:25:37,752 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:25:37,752 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:25:37,774 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:25:37,774 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:25:38,181 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 61 2017-08-28 15:25:38,185 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:38,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-08-28 15:25:48,204 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 62 2017-08-28 15:25:48,204 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:48,204 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-08-28 15:25:52,592 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:52,592 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:52,595 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:52,598 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:52,598 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:25:52,609 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:25:52,610 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:25:52,611 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:25:52,612 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:25:52,655 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:25:52,655 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:25:52,656 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:25:52,656 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:25:52,659 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:25:52,663 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:25:52,666 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:25:52,671 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:25:58,206 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 63 2017-08-28 15:25:58,223 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:25:58,224 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-08-28 15:26:06,978 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 58 2017-08-28 15:26:06,978 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:26:06,978 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 65 to run in 60 seconds 2017-08-28 15:26:06,991 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 9 2017-08-28 15:26:06,992 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:26:07,009 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-08-28 15:26:07,619 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:07,620 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:07,622 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:07,631 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:07,632 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:07,635 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:07,641 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45408384 2017-08-28 15:26:07,645 | TRACE | remote.py ( 84) | connect | connecting socket 78 45621376 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:07,646 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:26:07,646 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:07,658 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:07,658 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45722704 2017-08-28 15:26:07,683 | TRACE | remote.py ( 84) | connect | connecting socket 81 45408384 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:07,684 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:26:07,687 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:07,708 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:07,708 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45623280 2017-08-28 15:26:07,723 | TRACE | remote.py ( 84) | connect | connecting socket 85 45630416 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:07,724 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:26:07,724 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:07,727 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:07,732 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45624928 2017-08-28 15:26:07,741 | TRACE | remote.py ( 84) | connect | connecting socket 88 45706288 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:07,758 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:26:07,758 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:26:07,760 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.159 seconds 2017-08-28 15:26:07,783 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:26:07,830 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:26:07,836 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:26:07,836 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:26:07,843 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:26:07,846 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:26:07,852 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:26:07,852 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:26:08,225 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 64 2017-08-28 15:26:08,235 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:08,236 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 66 to run in 10 seconds 2017-08-28 15:26:18,254 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 66 2017-08-28 15:26:18,255 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:18,257 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-08-28 15:26:22,624 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:22,624 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:22,627 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:22,627 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:22,627 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:26:22,641 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:26:22,641 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:26:22,642 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:26:22,643 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:26:22,670 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:26:22,670 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:26:22,681 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:26:22,681 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:26:22,682 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:26:22,682 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:26:22,682 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:26:22,683 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:26:28,281 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 67 2017-08-28 15:26:28,281 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:28,281 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-08-28 15:26:37,631 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:37,632 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:37,633 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:37,634 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:37,634 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:37,634 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:37,636 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45621376 2017-08-28 15:26:37,650 | TRACE | remote.py ( 84) | connect | connecting socket 78 45623280 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:37,651 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:26:37,652 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:37,663 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:37,664 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45408384 2017-08-28 15:26:37,700 | TRACE | remote.py ( 84) | connect | connecting socket 81 45624928 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:37,702 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:26:37,703 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:37,703 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:37,717 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45630416 2017-08-28 15:26:37,717 | TRACE | remote.py ( 84) | connect | connecting socket 86 45408384 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:37,718 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:26:37,718 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:37,746 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:37,746 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45706288 2017-08-28 15:26:37,747 | TRACE | remote.py ( 84) | connect | connecting socket 87 45619984 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:37,763 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:26:37,776 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:26:37,777 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.164 seconds 2017-08-28 15:26:37,852 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:26:37,853 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:26:37,853 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:26:37,853 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:26:37,854 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:26:37,870 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:26:37,886 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:26:37,886 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:26:38,285 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 68 2017-08-28 15:26:38,307 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:38,315 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-08-28 15:26:48,331 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 69 2017-08-28 15:26:48,337 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:48,337 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-08-28 15:26:52,638 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:26:52,639 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:26:52,639 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:26:52,647 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:26:52,648 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:26:52,648 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:26:52,648 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:26:52,648 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:26:52,654 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:26:52,692 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:26:52,692 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:26:52,692 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:26:52,693 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:26:52,699 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:26:52,700 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:26:52,700 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:26:52,700 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:26:58,342 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 70 2017-08-28 15:26:58,356 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:26:58,356 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-08-28 15:27:06,981 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 65 2017-08-28 15:27:06,994 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:27:06,995 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 72 to run in 60 seconds 2017-08-28 15:27:07,014 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 10 2017-08-28 15:27:07,017 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:27:07,019 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-08-28 15:27:07,640 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:07,649 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:07,657 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:07,658 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:07,658 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:07,668 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:07,668 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45623280 2017-08-28 15:27:07,673 | TRACE | remote.py ( 84) | connect | connecting socket 78 45623280 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:07,677 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:27:07,677 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:07,678 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:07,679 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45624928 2017-08-28 15:27:07,701 | TRACE | remote.py ( 84) | connect | connecting socket 81 45624928 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:07,702 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:27:07,702 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:07,711 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:07,712 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45408384 2017-08-28 15:27:07,712 | TRACE | remote.py ( 84) | connect | connecting socket 84 45631472 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:07,712 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:27:07,712 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:07,713 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:07,713 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45619984 2017-08-28 15:27:07,754 | TRACE | remote.py ( 84) | connect | connecting socket 86 45408384 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:07,776 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:27:07,777 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:27:07,796 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.178 seconds 2017-08-28 15:27:07,839 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:27:07,856 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:27:07,857 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:27:07,857 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:27:07,857 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:27:07,858 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:27:07,858 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:27:07,858 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:27:08,361 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 71 2017-08-28 15:27:08,378 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:08,379 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 73 to run in 10 seconds 2017-08-28 15:27:18,398 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 73 2017-08-28 15:27:18,401 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:18,401 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-08-28 15:27:22,649 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:22,650 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:22,650 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:22,651 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:22,655 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:27:22,667 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:27:22,667 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:27:22,669 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:27:22,669 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:27:22,688 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:27:22,691 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:27:22,706 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:27:22,706 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:27:22,706 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:27:22,707 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:27:22,707 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:27:22,712 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:27:28,414 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 74 2017-08-28 15:27:28,423 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:28,430 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-08-28 15:27:37,660 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:37,661 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:37,661 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:37,661 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:37,662 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:37,662 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:37,662 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45623280 2017-08-28 15:27:37,695 | TRACE | remote.py ( 84) | connect | connecting socket 78 45623280 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:37,696 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:27:37,696 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:37,696 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:37,720 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45624928 2017-08-28 15:27:37,728 | TRACE | remote.py ( 84) | connect | connecting socket 87 45619984 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:37,729 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:27:37,740 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:37,740 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:37,741 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 84 45631472 2017-08-28 15:27:37,747 | TRACE | remote.py ( 84) | connect | connecting socket 88 45706288 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:37,755 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:27:37,756 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:37,758 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:37,766 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45408384 2017-08-28 15:27:37,771 | TRACE | remote.py ( 84) | connect | connecting socket 84 45624928 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:37,778 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:27:37,790 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:27:37,800 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.147 seconds 2017-08-28 15:27:37,843 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:27:37,859 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:27:37,859 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:27:37,860 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:27:37,860 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:27:37,862 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:27:37,863 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:27:37,863 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:27:38,432 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 75 2017-08-28 15:27:38,435 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:38,443 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-08-28 15:27:40,125 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.237 seconds 2017-08-28 15:27:48,462 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 76 2017-08-28 15:27:48,464 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:48,464 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 77 to run in 10 seconds 2017-08-28 15:27:52,670 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:27:52,684 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:27:52,685 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:27:52,685 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:27:52,685 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:27:52,691 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:27:52,691 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:27:52,692 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:27:52,692 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:27:52,736 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:27:52,736 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:27:52,743 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:27:52,744 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:27:52,745 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:27:52,753 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:27:52,754 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:27:52,754 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:27:58,476 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 77 2017-08-28 15:27:58,477 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:27:58,477 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-08-28 15:28:07,023 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 72 2017-08-28 15:28:07,024 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:28:07,024 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 79 to run in 60 seconds 2017-08-28 15:28:07,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 11 2017-08-28 15:28:07,024 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:28:07,025 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-08-28 15:28:07,682 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:07,688 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:07,693 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:07,693 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:07,705 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:07,706 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:07,706 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45623280 2017-08-28 15:28:07,707 | TRACE | remote.py ( 84) | connect | connecting socket 78 45623280 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:07,707 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:28:07,716 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:07,717 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:07,717 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 87 45619984 2017-08-28 15:28:07,753 | TRACE | remote.py ( 84) | connect | connecting socket 81 45409504 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:07,753 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:28:07,754 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:07,775 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:07,776 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 88 45706288 2017-08-28 15:28:07,776 | TRACE | remote.py ( 84) | connect | connecting socket 85 45630352 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:07,776 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:28:07,777 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:07,777 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:07,781 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 84 45624928 2017-08-28 15:28:07,784 | TRACE | remote.py ( 84) | connect | connecting socket 86 45632000 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:07,787 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:28:07,791 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:07,800 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.120 seconds 2017-08-28 15:28:07,853 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node1 2017-08-28 15:28:07,853 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node1 2017-08-28 15:28:07,854 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node3 2017-08-28 15:28:07,854 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node3 2017-08-28 15:28:07,854 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node2 2017-08-28 15:28:07,877 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node2 2017-08-28 15:28:07,877 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node4 2017-08-28 15:28:07,877 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node4 2017-08-28 15:28:08,485 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 78 2017-08-28 15:28:08,486 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:08,486 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 80 to run in 10 seconds 2017-08-28 15:28:10,486 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:28:10,519 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from Node2 2017-08-28 15:28:10,520 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged Node2 2017-08-28 15:28:10,520 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from Node3 2017-08-28 15:28:10,520 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged Node3 2017-08-28 15:28:10,521 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'po' to Node3 2017-08-28 15:28:10,521 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'po' to Node3 2017-08-28 15:28:10,521 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'po' to Node2 2017-08-28 15:28:10,522 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'po' to Node2 2017-08-28 15:28:10,562 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:10,573 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from Node1 2017-08-28 15:28:10,590 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged Node1 2017-08-28 15:28:10,618 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'po' to Node1 2017-08-28 15:28:10,619 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'po' to Node1 2017-08-28 15:28:10,630 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:28:10,658 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from Node4 2017-08-28 15:28:10,659 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged Node4 2017-08-28 15:28:10,659 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:28:10,666 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:28:10,679 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'po' to Node4 2017-08-28 15:28:10,681 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'po' to Node4 2017-08-28 15:28:10,681 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node6 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:28:10,681 | DEBUG | message_processor.py ( 29) | discard | Node6 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:28:10,704 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:28:10,704 | DEBUG | zstack.py ( 661) | handlePingPong | Node6C got ping from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:28:10,705 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'po' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:28:10,712 | DEBUG | zstack.py ( 643) | sendPingPong | Node6C ponged b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:28:10,735 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:28:10,749 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"ledgerId":0,"txnSeqNo":7,"viewNo":2,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"ledgerId":1,"txnSeqNo":15,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"ledgerId":2,"txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']} 2017-08-28 15:28:10,760 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"ledgerId":0,"txnSeqNo":7,"viewNo":2,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"ledgerId":1,"txnSeqNo":15,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"ledgerId":2,"txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:28:10,760 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"ledgerId":0,"txnSeqNo":7,"viewNo":2,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"ledgerId":1,"txnSeqNo":15,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"ledgerId":2,"txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']} 2017-08-28 15:28:10,761 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,779 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}], 'op': 'CURRENT_STATE'}, 'Node3') 2017-08-28 15:28:10,796 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,797 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,797 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'txnSeqNo': 7, 'viewNo': 2, 'ppSeqNo': 6, 'ledgerId': 0, 'op': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:10,808 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,808 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,808 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ppSeqNo': 6, 'ledgerId': 1, 'op': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:10,815 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,815 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,816 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'op': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:10,821 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,821 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}]}', '{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"viewNo":2,"ledgerId":0,"ppSeqNo":6}', '{"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"viewNo":0,"ledgerId":1,"ppSeqNo":6}', '{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"viewNo":null,"ledgerId":2,"ppSeqNo":null}']} 2017-08-28 15:28:10,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}]}', '{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"viewNo":2,"ledgerId":0,"ppSeqNo":6}', '{"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"viewNo":0,"ledgerId":1,"ppSeqNo":6}', '{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"viewNo":null,"ledgerId":2,"ppSeqNo":null}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:28:10,823 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}]}', '{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"viewNo":2,"ledgerId":0,"ppSeqNo":6}', '{"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"viewNo":0,"ledgerId":1,"ppSeqNo":6}', '{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"viewNo":null,"ledgerId":2,"ppSeqNo":null}']} 2017-08-28 15:28:10,824 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,846 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}], 'op': 'CURRENT_STATE'}, 'Node2') 2017-08-28 15:28:10,846 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,847 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,847 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'txnSeqNo': 7, 'viewNo': 2, 'ppSeqNo': 6, 'ledgerId': 0, 'op': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:10,848 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,850 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,852 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ppSeqNo': 6, 'ledgerId': 1, 'op': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:10,853 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,853 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,853 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'op': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:10,853 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,870 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"ledgerId":0,"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","txnSeqNo":15,"ledgerId":1,"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2,"viewNo":null}']} 2017-08-28 15:28:10,871 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"ledgerId":0,"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","txnSeqNo":15,"ledgerId":1,"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2,"viewNo":null}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:28:10,875 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"ledgerId":0,"viewNo":2}', '{"op":"LEDGER_STATUS","ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","txnSeqNo":15,"ledgerId":1,"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2,"viewNo":null}']} 2017-08-28 15:28:10,877 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,880 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}], 'op': 'CURRENT_STATE'}, 'Node1') 2017-08-28 15:28:10,882 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:10,883 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,883 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'txnSeqNo': 7, 'viewNo': 2, 'ppSeqNo': 6, 'ledgerId': 0, 'op': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:10,886 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:10,903 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,907 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ppSeqNo': 6, 'ledgerId': 1, 'op': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:10,908 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:10,909 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,910 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'op': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:10,912 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:10,912 | DEBUG | node.py ( 889) | process_current_state_message | Node6 processing current state CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} from Node3 2017-08-28 15:28:10,924 | DEBUG | node.py (1152) | msgHasAcceptableViewNo | Node6 stashing a message for a future view: VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:10,929 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from Node3 2017-08-28 15:28:10,929 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:10,930 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:10,930 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node3'] 2017-08-28 15:28:10,934 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node3 2017-08-28 15:28:10,948 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node3 2017-08-28 15:28:10,948 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node3 2017-08-28 15:28:10,952 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node3 2017-08-28 15:28:10,958 | DEBUG | node.py ( 889) | process_current_state_message | Node6 processing current state CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} from Node2 2017-08-28 15:28:10,959 | DEBUG | node.py (1152) | msgHasAcceptableViewNo | Node6 stashing a message for a future view: VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:10,963 | INFO | node.py (1948) | _start_view_change_if_possible | VIEW CHANGE: Node6 starting view change for 2 after 2 view change indications from other nodes 2017-08-28 15:28:10,964 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action _check_view_change_completed with id 81 to run in 60 seconds 2017-08-28 15:28:10,964 | DEBUG | replica.py ( 453) | on_view_change_start | Node6:0 setting last prepared for master to None 2017-08-28 15:28:10,964 | DEBUG | node.py (2127) | startViewChange | Node6 resetting monitor stats after view change 2017-08-28 15:28:10,964 | DEBUG | monitor.py ( 192) | reset | Node6's Monitor being reset 2017-08-28 15:28:10,965 | DEBUG | node.py (1187) | sendToElector | Node6 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}, 'Node3') 2017-08-28 15:28:10,965 | DEBUG | node.py (1187) | sendToElector | Node6 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}, 'Node2') 2017-08-28 15:28:10,965 | DEBUG | node.py ( 978) | processStashedMsgsForView | Node6 processed 2 stashed msgs for view no 2 2017-08-28 15:28:10,982 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node6 changed to view 2, will start catchup now 2017-08-28 15:28:10,985 | DEBUG | node.py (2166) | start_catchup | Node6 does not start the catchup procedure because it is already in this state 2017-08-28 15:28:10,986 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from Node2 2017-08-28 15:28:10,988 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:10,990 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,007 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node2'] 2017-08-28 15:28:11,008 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node2 2017-08-28 15:28:11,008 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node2 2017-08-28 15:28:11,010 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node2 2017-08-28 15:28:11,013 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node2 2017-08-28 15:28:11,017 | DEBUG | node.py ( 889) | process_current_state_message | Node6 processing current state CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} from Node1 2017-08-28 15:28:11,017 | DEBUG | node.py (1187) | sendToElector | Node6 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}, 'Node1') 2017-08-28 15:28:11,018 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from Node1 2017-08-28 15:28:11,018 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,018 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,019 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node1'] 2017-08-28 15:28:11,019 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node1 2017-08-28 15:28:11,020 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node1 2017-08-28 15:28:11,020 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node1 2017-08-28 15:28:11,021 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node1 2017-08-28 15:28:11,022 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:28:11,022 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:11,023 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:11,046 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:28:11,048 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,048 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,048 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,049 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:28:11,052 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node1 2017-08-28 15:28:11,052 | DEBUG | zstack.py ( 726) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,052 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node1 2017-08-28 15:28:11,074 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node3 2017-08-28 15:28:11,074 | DEBUG | zstack.py ( 726) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,075 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node3 2017-08-28 15:28:11,076 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node2 2017-08-28 15:28:11,077 | DEBUG | zstack.py ( 726) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,077 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node2 2017-08-28 15:28:11,092 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:11,093 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2,"op":"CURRENT_STATE"}', '{"txnSeqNo":7,"ledgerId":0,"ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2,"op":"LEDGER_STATUS"}', '{"txnSeqNo":15,"ledgerId":1,"ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"LEDGER_STATUS"}', '{"txnSeqNo":0,"ledgerId":2,"ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:28:11,104 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2,"op":"CURRENT_STATE"}', '{"txnSeqNo":7,"ledgerId":0,"ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2,"op":"LEDGER_STATUS"}', '{"txnSeqNo":15,"ledgerId":1,"ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"LEDGER_STATUS"}', '{"txnSeqNo":0,"ledgerId":2,"ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","viewNo":null,"op":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:28:11,104 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"viewNo":2,"op":"CURRENT_STATE"}', '{"txnSeqNo":7,"ledgerId":0,"ppSeqNo":6,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2,"op":"LEDGER_STATUS"}', '{"txnSeqNo":15,"ledgerId":1,"ppSeqNo":6,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"LEDGER_STATUS"}', '{"txnSeqNo":0,"ledgerId":2,"ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:28:11,115 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:11,116 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}], 'op': 'CURRENT_STATE'}, 'Node4') 2017-08-28 15:28:11,123 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:11,124 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:11,124 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'txnSeqNo': 7, 'viewNo': 2, 'ppSeqNo': 6, 'ledgerId': 0, 'op': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:28:11,127 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:11,132 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:11,132 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ppSeqNo': 6, 'ledgerId': 1, 'op': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:28:11,132 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:11,141 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:11,142 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'op': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:28:11,142 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:11,156 | DEBUG | node.py ( 889) | process_current_state_message | Node6 processing current state CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} from Node4 2017-08-28 15:28:11,159 | DEBUG | node.py (1187) | sendToElector | Node6 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}, 'Node4') 2017-08-28 15:28:11,160 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from Node4 2017-08-28 15:28:11,164 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,164 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 4 with 7 2017-08-28 15:28:11,167 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node4'] 2017-08-28 15:28:11,171 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node4 2017-08-28 15:28:11,172 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node4 2017-08-28 15:28:11,172 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node4 2017-08-28 15:28:11,176 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node6 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node4 2017-08-28 15:28:11,190 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node4 2017-08-28 15:28:11,191 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,200 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}' to Node4 2017-08-28 15:28:11,202 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:28:11,216 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,219 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'op': 'CONSISTENCY_PROOF'}, 'Node2') 2017-08-28 15:28:11,224 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,225 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,225 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'op': 'CONSISTENCY_PROOF'}, 'Node3') 2017-08-28 15:28:11,227 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,236 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,236 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'op': 'CONSISTENCY_PROOF'}, 'Node1') 2017-08-28 15:28:11,243 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,248 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} from Node2 2017-08-28 15:28:11,249 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node6 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-08-28 15:28:11,253 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} from Node3 2017-08-28 15:28:11,254 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action partial(request_CPs_if_needed) with id 1 to run in 20 seconds 2017-08-28 15:28:11,268 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node6 deciding on the basis of CPs {'Node3': CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0}, 'Node2': CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0}} and f 1 2017-08-28 15:28:11,269 | DEBUG | node.py (1461) | preLedgerCatchUp | Node6 going to process any ordered requests before starting catchup. 2017-08-28 15:28:11,269 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:11,270 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:28:11,279 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:11,280 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:28:11,280 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node6 processed 0 stashed ordered requests 2017-08-28 15:28:11,280 | DEBUG | monitor.py ( 192) | reset | Node6's Monitor being reset 2017-08-28 15:28:11,282 | INFO | node.py (1471) | preLedgerCatchUp | Node6 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:28:11,283 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node6 started catching up with consistency proof CONSISTENCY_PROOF{'hashes': ('DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q',), 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,284 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 7, 'seqNoStart': 5, 'ledgerId': 0, 'seqNoEnd': 5} to 1 recipients: ['Node1'] 2017-08-28 15:28:11,287 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 7, 'seqNoStart': 6, 'ledgerId': 0, 'seqNoEnd': 6} to 1 recipients: ['Node2'] 2017-08-28 15:28:11,288 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 7, 'seqNoStart': 7, 'ledgerId': 0, 'seqNoEnd': 7} to 1 recipients: ['Node3'] 2017-08-28 15:28:11,303 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action partial(request_txns_if_needed) with id 2 to run in 15.299999999999999 seconds 2017-08-28 15:28:11,311 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} from Node1 2017-08-28 15:28:11,312 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:11,323 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"catchupTill":7,"seqNoStart":5,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":5}' to Node1 2017-08-28 15:28:11,329 | DEBUG | zstack.py ( 726) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,330 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"catchupTill":7,"seqNoStart":5,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":5}' to Node1 2017-08-28 15:28:11,330 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"catchupTill":7,"seqNoStart":7,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":7}' to Node3 2017-08-28 15:28:11,331 | DEBUG | zstack.py ( 726) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,334 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"catchupTill":7,"seqNoStart":7,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":7}' to Node3 2017-08-28 15:28:11,338 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"catchupTill":7,"seqNoStart":6,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":6}' to Node2 2017-08-28 15:28:11,338 | DEBUG | zstack.py ( 726) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:11,339 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"catchupTill":7,"seqNoStart":6,"ledgerId":0,"op":"CATCHUP_REQ","seqNoEnd":6}' to Node2 2017-08-28 15:28:11,352 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:11,353 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,368 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'op': 'CONSISTENCY_PROOF'}, 'Node4') 2017-08-28 15:28:11,368 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} 2017-08-28 15:28:11,369 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['DNjKvRRd7MQg4SM2yfQUrYwS6akgsWoerDTr6Rc4kF2q'], 'seqNoStart': 4, 'seqNoEnd': 7, 'viewNo': 2, 'newMerkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0} from Node4 2017-08-28 15:28:11,380 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:11,408 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.923 seconds 2017-08-28 15:28:11,424 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:28:11,426 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CATCHUP_REP{'txns': {'6': {'signature': '4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT', 'reqId': 1503934008154939, 'type': '0', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1503934008}}, 'ledgerId': 0, 'consProof': ['5yYbh3Csn1bGm21gsiv6vuQBMZg8dedahhTCWvr7WPk2', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,441 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'6': {'signature': '4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT', 'reqId': 1503934008154939, 'type': '0', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1503934008}}, 'ledgerId': 0, 'op': 'CATCHUP_REP', 'consProof': ['5yYbh3Csn1bGm21gsiv6vuQBMZg8dedahhTCWvr7WPk2', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']}, 'Node2') 2017-08-28 15:28:11,441 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'6': {'signature': '4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT', 'reqId': 1503934008154939, 'type': '0', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1503934008}}, 'ledgerId': 0, 'consProof': ['5yYbh3Csn1bGm21gsiv6vuQBMZg8dedahhTCWvr7WPk2', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,443 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CATCHUP_REP{'txns': {'7': {'signature': 'zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ', 'reqId': 1503934088807442, 'txnTime': 1503934089, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd', 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': ['VALIDATOR']}, 'type': '0'}}, 'ledgerId': 0, 'consProof': []} 2017-08-28 15:28:11,453 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'7': {'signature': 'zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ', 'reqId': 1503934088807442, 'txnTime': 1503934089, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd', 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': ['VALIDATOR']}, 'type': '0'}}, 'ledgerId': 0, 'op': 'CATCHUP_REP', 'consProof': []}, 'Node3') 2017-08-28 15:28:11,454 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'7': {'signature': 'zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ', 'reqId': 1503934088807442, 'txnTime': 1503934089, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd', 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': ['VALIDATOR']}, 'type': '0'}}, 'ledgerId': 0, 'consProof': []} 2017-08-28 15:28:11,455 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CATCHUP_REP{'txns': {'5': {'signature': '5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA', 'reqId': 1503933979830988, 'txnTime': 1503933980, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105', 'node_port': 9701}, 'type': '0'}}, 'ledgerId': 0, 'consProof': ['Etup9NgC9tEUzHq8fzn8VAZ6YyTW7YyYvrvYYtRZbwdj', '4K3SsKB7U1ZgzjLhZubLH7qJmEKWBQdGmPXCEGAEHP1g', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,471 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'5': {'signature': '5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA', 'reqId': 1503933979830988, 'txnTime': 1503933980, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105', 'node_port': 9701}, 'type': '0'}}, 'ledgerId': 0, 'op': 'CATCHUP_REP', 'consProof': ['Etup9NgC9tEUzHq8fzn8VAZ6YyTW7YyYvrvYYtRZbwdj', '4K3SsKB7U1ZgzjLhZubLH7qJmEKWBQdGmPXCEGAEHP1g', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']}, 'Node1') 2017-08-28 15:28:11,479 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'5': {'signature': '5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA', 'reqId': 1503933979830988, 'txnTime': 1503933980, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105', 'node_port': 9701}, 'type': '0'}}, 'ledgerId': 0, 'consProof': ['Etup9NgC9tEUzHq8fzn8VAZ6YyTW7YyYvrvYYtRZbwdj', '4K3SsKB7U1ZgzjLhZubLH7qJmEKWBQdGmPXCEGAEHP1g', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,481 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node2: CATCHUP_REP{'txns': {'6': {'signature': '4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT', 'reqId': 1503934008154939, 'type': '0', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1503934008}}, 'ledgerId': 0, 'consProof': ['5yYbh3Csn1bGm21gsiv6vuQBMZg8dedahhTCWvr7WPk2', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,487 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 1 transactions in the catchup from Node2 2017-08-28 15:28:11,495 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:11,496 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 1 of them now, from 6 to 6 2017-08-28 15:28:11,496 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 0 catchup replies with sequence numbers [] 2017-08-28 15:28:11,499 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node3: CATCHUP_REP{'txns': {'7': {'signature': 'zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ', 'reqId': 1503934088807442, 'txnTime': 1503934089, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd', 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': ['VALIDATOR']}, 'type': '0'}}, 'ledgerId': 0, 'consProof': []} 2017-08-28 15:28:11,505 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 1 transactions in the catchup from Node3 2017-08-28 15:28:11,507 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:11,518 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 2 of them now, from 6 to 7 2017-08-28 15:28:11,519 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 0 catchup replies with sequence numbers [] 2017-08-28 15:28:11,525 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node1: CATCHUP_REP{'txns': {'5': {'signature': '5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA', 'reqId': 1503933979830988, 'txnTime': 1503933980, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105', 'node_port': 9701}, 'type': '0'}}, 'ledgerId': 0, 'consProof': ['Etup9NgC9tEUzHq8fzn8VAZ6YyTW7YyYvrvYYtRZbwdj', '4K3SsKB7U1ZgzjLhZubLH7qJmEKWBQdGmPXCEGAEHP1g', 'GMnnuWPjRegPoQDBTWCYX5rJEvo6y4odvZmf3yabHbLM', '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3']} 2017-08-28 15:28:11,526 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 1 transactions in the catchup from Node1 2017-08-28 15:28:11,527 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:11,527 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 3 of them now, from 5 to 7 2017-08-28 15:28:11,528 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 5, 7, b'v\xe7n\x85\xea\xa0{[t\x194T\xb6h\x01S\xcf\xe7\xc5Z\xc3N5U\x15\xee\xf0\xf7#\xc2\xc3\xd9', b'K\xfc\xa00\x88\x15-\x1f\xe3O\xd4?Z\x86\xa7\x9a\xd0L\xa5=6\x06`\xae[67a\xb8v\x00\xd3', [b'\xcetDW\xf6\xfe}\xf0\xfe\x0c\x93\xfd7\x11\xc3\xd3\xedv\xa6\xa1\x18!\xddYO\xe9\xa7&\x1e\x99\x11z', b'12G\xf1\xc9i^\xb0Z\xcayG`f\xea\xf5\x07<0@\xd6m\xec\x16\xcb\xfc\x8b\xdd)w\xb5o', b'\xe42\xb2X&\x1f\x1f\\\xf9\x83\xba%\xddj,\x142\xb7A\xa5A\xe5\xc4\xfbz\xbb\x84"\xd7\x81dj', b"I\xb5*\x8a&f\x9e\x95\x8a\x93\xcc8\x0b\xfc\xb6\xfa\x9b\xd3R\x0e\xef\x87'\xba\x99\x89\xb2\x00H\x99m\xbe"] 2017-08-28 15:28:11,563 | DEBUG | stack_manager.py ( 160) | connectNewRemote | Node6 adding new node Node5 with HA ('10.0.0.105', 9701) 2017-08-28 15:28:11,563 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:11,564 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:11,565 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:11,565 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:11,567 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node6 found the following missing connections: Node5 2017-08-28 15:28:11,583 | TRACE | remote.py ( 84) | connect | connecting socket 93 45716448 to remote Node5:HA(host='10.0.0.105', port=9701) 2017-08-28 15:28:11,583 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node5 at 10.0.0.105:9701 2017-08-28 15:28:11,584 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node5 2017-08-28 15:28:11,584 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:28:11,586 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:28:11,587 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:28:11,591 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:28:11,591 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:11,593 | DEBUG | node.py ( 861) | sendPoolInfoToClients | Node6 sending new node info {'signature': '5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA', 'reqId': 1503933979830988, 'txnTime': 1503933980, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105', 'node_port': 9701}, 'type': '0', 'seqNo': 5} to all clients 2017-08-28 15:28:11,594 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"txn":{"signature":"5odcUT8Bzxrau3HGCBQ71nGHPmZa7SJmWhzTWT4BHHxL6mxYgUfoH6mahDvRQwSE1gnGNsUgqci23WXwQLehiDAA","reqId":1503933979830988,"txnTime":1503933980,"identifier":"XhYtvJqezMUKfF6KVNaGmT","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"client_ip":"10.0.0.105","node_ip":"10.0.0.105","node_port":9701},"type":"0","seqNo":5},"op":"POOL_LEDGER_TXNS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:28:11,617 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 6, 7, b'_G\x046\x82\x17\x10\x98~\xc0b\x0b\xa3n\xb0\xe6\xd2\xb8\xdd\xe5\xf5\xe5\xb7s\xc2\xfe\xcaF>\x08\xfd\xa7', b'K\xfc\xa00\x88\x15-\x1f\xe3O\xd4?Z\x86\xa7\x9a\xd0L\xa5=6\x06`\xae[67a\xb8v\x00\xd3', [b'I\xeb\x00H\xb0@\x89Vy\xc4\xe4\x87n\xf2m_N\x18\xc6\x0e\xf7\xae\xd4Yv"ey\xcaE\x0e\x1f', b'\xe42\xb2X&\x1f\x1f\\\xf9\x83\xba%\xddj,\x142\xb7A\xa5A\xe5\xc4\xfbz\xbb\x84"\xd7\x81dj', b"I\xb5*\x8a&f\x9e\x95\x8a\x93\xcc8\x0b\xfc\xb6\xfa\x9b\xd3R\x0e\xef\x87'\xba\x99\x89\xb2\x00H\x99m\xbe"] 2017-08-28 15:28:11,643 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node5:HA(host='10.0.0.105', port=9701) 2017-08-28 15:28:11,645 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 93 45716448 2017-08-28 15:28:11,646 | DEBUG | stack_manager.py ( 212) | removeRemote | Node6 removed remote Node5 2017-08-28 15:28:11,646 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:11,646 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:11,646 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:11,647 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:11,663 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node6 found the following missing connections: Node5 2017-08-28 15:28:11,664 | TRACE | remote.py ( 84) | connect | connecting socket 93 45716448 to remote Node5:HA(host='10.0.0.6', port=9701) 2017-08-28 15:28:11,664 | INFO | zstack.py ( 588) | connect | CONNECTION: Node6 looking for Node5 at 10.0.0.6:9701 2017-08-28 15:28:11,667 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node5 2017-08-28 15:28:11,667 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:11,668 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:11,668 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 45623280 2017-08-28 15:28:11,673 | TRACE | remote.py ( 84) | connect | connecting socket 78 45623280 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:11,674 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node1 2017-08-28 15:28:11,709 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:11,710 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:11,710 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 81 45409504 2017-08-28 15:28:11,711 | TRACE | remote.py ( 84) | connect | connecting socket 95 45765984 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:11,712 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node3 2017-08-28 15:28:11,712 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:11,716 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:11,716 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 85 45630352 2017-08-28 15:28:11,717 | TRACE | remote.py ( 84) | connect | connecting socket 96 45800464 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:11,742 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node2 2017-08-28 15:28:11,743 | DEBUG | zstack.py ( 603) | reconnectRemote | Node6 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:11,751 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:11,755 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 86 45632000 2017-08-28 15:28:11,756 | TRACE | remote.py ( 84) | connect | connecting socket 97 45552320 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:11,759 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node4 2017-08-28 15:28:11,759 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:11,769 | DEBUG | pool_manager.py ( 298) | nodeServicesChanged | Node 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc not changing services since it is same as existing 2017-08-28 15:28:11,770 | DEBUG | node.py ( 861) | sendPoolInfoToClients | Node6 sending new node info {'signature': '4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT', 'reqId': 1503934008154939, 'type': '0', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': ['VALIDATOR'], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1503934008, 'seqNo': 6} to all clients 2017-08-28 15:28:11,785 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"txn":{"signature":"4JursX6ywfiH2shqqBEXQX5r852y5szJUBZ8YWSeLeueC3jHK6pUGTqJ3mVrw76g1RzPxcJbtM2wMjnT9Ahp8VbT","reqId":1503934008154939,"type":"0","identifier":"XhYtvJqezMUKfF6KVNaGmT","data":{"client_port":9702,"node_ip":"10.0.0.6","services":["VALIDATOR"],"client_ip":"10.0.0.6","alias":"Node5","node_port":9701},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","txnTime":1503934008,"seqNo":6},"op":"POOL_LEDGER_TXNS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:28:11,793 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 7, 7, b'K\xfc\xa00\x88\x15-\x1f\xe3O\xd4?Z\x86\xa7\x9a\xd0L\xa5=6\x06`\xae[67a\xb8v\x00\xd3', b'K\xfc\xa00\x88\x15-\x1f\xe3O\xd4?Z\x86\xa7\x9a\xd0L\xa5=6\x06`\xae[67a\xb8v\x00\xd3', [] 2017-08-28 15:28:11,832 | DEBUG | pool_manager.py ( 298) | nodeServicesChanged | Node 6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G not changing services since it is same as existing 2017-08-28 15:28:11,843 | DEBUG | node.py ( 861) | sendPoolInfoToClients | Node6 sending new node info {'signature': 'zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ', 'reqId': 1503934088807442, 'txnTime': 1503934089, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd', 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': ['VALIDATOR']}, 'type': '0', 'seqNo': 7} to all clients 2017-08-28 15:28:11,844 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"txn":{"signature":"zhfYef2NnAS7vCPvewudttE57nFzERPnjyxvcerkg6x5fXuqvy81kN3uLESCSJjuVNRqs8yhfq2wGbSJbWa2odQ","reqId":1503934088807442,"txnTime":1503934089,"identifier":"FBQ5W7LxRp8bLeVNEafjpd","dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G","data":{"client_port":9702,"alias":"Node6","node_port":9701,"client_ip":"10.0.0.7","node_ip":"10.0.0.7","services":["VALIDATOR"]},"type":"0","seqNo":7},"op":"POOL_LEDGER_TXNS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:28:11,876 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 3 catchup replies with sequence numbers [5, 6, 7] 2017-08-28 15:28:11,878 | DEBUG | node.py ( 923) | checkInstances | Node6 choosing to start election on the basis of count 1 and nodes set() 2017-08-28 15:28:11,881 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:28:11,881 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node1 for ledger status of ledger 2 2017-08-28 15:28:11,881 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:28:11,882 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node2 for ledger status of ledger 2 2017-08-28 15:28:11,921 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:28:11,921 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node3 for ledger status of ledger 2 2017-08-28 15:28:11,922 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:28:11,928 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node4 for ledger status of ledger 2 2017-08-28 15:28:11,929 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:28:11,929 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node6 for ledger status of ledger 2 2017-08-28 15:28:11,930 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node5'] 2017-08-28 15:28:11,939 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node5 for ledger status of ledger 2 2017-08-28 15:28:11,939 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node6 going to process 4 stashed ledger statuses for ledger 2 2017-08-28 15:28:11,943 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node3 2017-08-28 15:28:11,944 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:11,961 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:11,962 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node2 2017-08-28 15:28:11,962 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:11,964 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:11,964 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node1 2017-08-28 15:28:11,980 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:11,996 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,000 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node4 2017-08-28 15:28:12,006 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,014 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,015 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node1', 'Node3', 'Node2', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:28:12,015 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node6 found from ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} that it does not need catchup 2017-08-28 15:28:12,023 | DEBUG | node.py (1461) | preLedgerCatchUp | Node6 going to process any ordered requests before starting catchup. 2017-08-28 15:28:12,027 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:12,028 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:28:12,028 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:12,028 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:28:12,029 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node6 processed 0 stashed ordered requests 2017-08-28 15:28:12,029 | DEBUG | monitor.py ( 192) | reset | Node6's Monitor being reset 2017-08-28 15:28:12,031 | INFO | node.py (1471) | preLedgerCatchUp | Node6 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:28:12,037 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,038 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:28:12,038 | DEBUG | upgrader.py ( 208) | processLedger | 6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G processing config ledger for any upgrades 2017-08-28 15:28:12,038 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:28:12,038 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node1 for ledger status of ledger 1 2017-08-28 15:28:12,039 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:28:12,039 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node2 for ledger status of ledger 1 2017-08-28 15:28:12,039 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:28:12,039 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node3 for ledger status of ledger 1 2017-08-28 15:28:12,099 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:28:12,101 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node4 for ledger status of ledger 1 2017-08-28 15:28:12,101 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:28:12,102 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node6 for ledger status of ledger 1 2017-08-28 15:28:12,108 | DEBUG | node.py (2566) | send | Node6 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node5'] 2017-08-28 15:28:12,108 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node6 asking Node5 for ledger status of ledger 1 2017-08-28 15:28:12,108 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node6 going to process 4 stashed ledger statuses for ledger 1 2017-08-28 15:28:12,108 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node3 2017-08-28 15:28:12,112 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,114 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,120 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node3'] 2017-08-28 15:28:12,120 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node2 2017-08-28 15:28:12,128 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,128 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,129 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node2'] 2017-08-28 15:28:12,144 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node1 2017-08-28 15:28:12,145 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,145 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,166 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node1'] 2017-08-28 15:28:12,167 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node4 2017-08-28 15:28:12,167 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,172 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,172 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node4'] 2017-08-28 15:28:12,196 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node6' last upgrade txn is None 2017-08-28 15:28:12,196 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node6 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:28:12,200 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node6 completed catching up ledger 0, caught up 3 in total 2017-08-28 15:28:12,202 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 4 msgs to Node5 into one transmission 2017-08-28 15:28:12,205 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']) 2017-08-28 15:28:12,206 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,207 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH"}' to Node5 2017-08-28 15:28:12,222 | DEBUG | zstack.py ( 726) | transmit | Remote Node5 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,222 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 7 msgs to Node4 into one transmission 2017-08-28 15:28:12,223 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'pi', b'pi', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:12,232 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,232 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node4 2017-08-28 15:28:12,233 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,238 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 7 msgs to Node1 into one transmission 2017-08-28 15:28:12,238 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'pi', b'pi', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:12,244 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,255 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node1 2017-08-28 15:28:12,256 | DEBUG | zstack.py ( 726) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,256 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 7 msgs to Node3 into one transmission 2017-08-28 15:28:12,259 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'pi', b'pi', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:12,260 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node3: b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,274 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node3 2017-08-28 15:28:12,280 | DEBUG | zstack.py ( 726) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,281 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 7 msgs to Node2 into one transmission 2017-08-28 15:28:12,281 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'pi', b'pi', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:12,289 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node2: b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,297 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["pi","pi","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node2 2017-08-28 15:28:12,301 | DEBUG | zstack.py ( 726) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,301 | DEBUG | looper.py ( 212) | runOnceNicely | it took 0.882 seconds to run once nicely 2017-08-28 15:28:12,318 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.899 seconds 2017-08-28 15:28:12,367 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:12,368 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:28:12,368 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:28:12,369 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":null,"txnSeqNo":15,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:28:12,369 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,369 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:12,369 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,370 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,370 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:12,370 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,370 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,370 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:12,370 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,371 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,371 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:12,396 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,416 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node5 2017-08-28 15:28:12,417 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,428 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,428 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node5 2017-08-28 15:28:12,429 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,436 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,436 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node5 2017-08-28 15:28:12,444 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,445 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,445 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node5'] 2017-08-28 15:28:12,456 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node5 2017-08-28 15:28:12,456 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,456 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,458 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node5'] 2017-08-28 15:28:12,470 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node5 into one transmission 2017-08-28 15:28:12,470 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:12,471 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:12,474 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node5 2017-08-28 15:28:12,488 | DEBUG | zstack.py ( 726) | transmit | Remote Node5 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:12,489 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:28:12,502 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node4 2017-08-28 15:28:12,503 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node4 2017-08-28 15:28:12,505 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}']} 2017-08-28 15:28:12,507 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:28:12,519 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}']} 2017-08-28 15:28:12,520 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,532 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}}, 'Node4') 2017-08-28 15:28:12,532 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,532 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,539 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}}, 'Node4') 2017-08-28 15:28:12,539 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,540 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,540 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}}, 'Node4') 2017-08-28 15:28:12,540 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,550 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,551 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}}, 'Node4') 2017-08-28 15:28:12,562 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,562 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,563 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node4') 2017-08-28 15:28:12,568 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,568 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node1 2017-08-28 15:28:12,581 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node1 2017-08-28 15:28:12,582 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}']} 2017-08-28 15:28:12,582 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:28:12,598 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}']} 2017-08-28 15:28:12,599 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,602 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:12,603 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,604 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,604 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:12,605 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,605 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,605 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:12,621 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,621 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,622 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:28:12,622 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,622 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,626 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node1') 2017-08-28 15:28:12,627 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,628 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node3 2017-08-28 15:28:12,628 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node3 2017-08-28 15:28:12,629 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"seqNoEnd":15,"seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ledgerId":1,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6}']} 2017-08-28 15:28:12,651 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"seqNoEnd":15,"seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ledgerId":1,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:28:12,654 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"seqNoEnd":15,"seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ledgerId":1,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6}']} 2017-08-28 15:28:12,655 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,655 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:12,666 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,670 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,670 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:12,671 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,680 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,681 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:12,681 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,682 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,683 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-08-28 15:28:12,683 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,688 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,689 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node3') 2017-08-28 15:28:12,707 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,708 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node2 2017-08-28 15:28:12,708 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node2 2017-08-28 15:28:12,709 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"CONSISTENCY_PROOF","viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"ppSeqNo":6,"seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ledgerId":1}']} 2017-08-28 15:28:12,723 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"CONSISTENCY_PROOF","viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"ppSeqNo":6,"seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ledgerId":1}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:28:12,726 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"CONSISTENCY_PROOF","viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"ppSeqNo":6,"seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ledgerId":1}']} 2017-08-28 15:28:12,734 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,735 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:12,735 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,743 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,744 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:12,744 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0}} 2017-08-28 15:28:12,752 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,752 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:12,752 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,753 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,753 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:28:12,755 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15}} 2017-08-28 15:28:12,760 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,763 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node2') 2017-08-28 15:28:12,775 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,787 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node4 2017-08-28 15:28:12,788 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,788 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,797 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node4 2017-08-28 15:28:12,797 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,797 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,798 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node4 2017-08-28 15:28:12,800 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,820 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,820 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node4'] 2017-08-28 15:28:12,821 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node4 2017-08-28 15:28:12,821 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,822 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,823 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node4'] 2017-08-28 15:28:12,832 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node4 2017-08-28 15:28:12,839 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node6 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-08-28 15:28:12,850 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node1 2017-08-28 15:28:12,850 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,851 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,852 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node1 2017-08-28 15:28:12,854 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,854 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,855 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node1 2017-08-28 15:28:12,855 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,856 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,880 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node1'] 2017-08-28 15:28:12,880 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node1 2017-08-28 15:28:12,881 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,881 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,884 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node1'] 2017-08-28 15:28:12,885 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node1 2017-08-28 15:28:12,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action partial(request_CPs_if_needed) with id 3 to run in 25 seconds 2017-08-28 15:28:12,905 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node6 deciding on the basis of CPs {'Node1': CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1}, 'Node4': CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1}} and f 1 2017-08-28 15:28:12,908 | DEBUG | node.py (1461) | preLedgerCatchUp | Node6 going to process any ordered requests before starting catchup. 2017-08-28 15:28:12,911 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:12,912 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:28:12,913 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:12,927 | DEBUG | node.py (1869) | force_process_ordered | Node6 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:28:12,931 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node6 processed 0 stashed ordered requests 2017-08-28 15:28:12,931 | DEBUG | monitor.py ( 192) | reset | Node6's Monitor being reset 2017-08-28 15:28:12,932 | INFO | node.py (1471) | preLedgerCatchUp | Node6 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:28:12,932 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node6 started catching up with consistency proof CONSISTENCY_PROOF{'hashes': ('jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'), 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:12,934 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 15, 'seqNoStart': 10, 'ledgerId': 1, 'seqNoEnd': 11} to 1 recipients: ['Node1'] 2017-08-28 15:28:12,937 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 15, 'seqNoStart': 12, 'ledgerId': 1, 'seqNoEnd': 13} to 1 recipients: ['Node2'] 2017-08-28 15:28:12,938 | DEBUG | node.py (2566) | send | Node6 sending message CATCHUP_REQ{'catchupTill': 15, 'seqNoStart': 14, 'ledgerId': 1, 'seqNoEnd': 15} to 1 recipients: ['Node3'] 2017-08-28 15:28:12,938 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action partial(request_txns_if_needed) with id 4 to run in 15.600000000000001 seconds 2017-08-28 15:28:12,939 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node3 2017-08-28 15:28:12,942 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,942 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,943 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node5', 'Node4', 'Node3', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:28:12,944 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node3 2017-08-28 15:28:12,968 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,973 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:12,973 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node5', 'Node4', 'Node3', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:28:12,974 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node3 2017-08-28 15:28:12,976 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,990 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,991 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node3 2017-08-28 15:28:12,995 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,995 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:12,996 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node3 2017-08-28 15:28:12,997 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,015 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node2 2017-08-28 15:28:13,019 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:13,019 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:13,020 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node5', 'Node4', 'Node3', 'Node1', 'Node2'} that its ledger of type 2 is latest 2017-08-28 15:28:13,020 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node2 2017-08-28 15:28:13,024 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:13,039 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:13,039 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node5', 'Node4', 'Node3', 'Node1', 'Node2'} that its ledger of type 2 is latest 2017-08-28 15:28:13,040 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node2 2017-08-28 15:28:13,041 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:13,042 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:13,057 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 0, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node2 2017-08-28 15:28:13,057 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:13,058 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 9 with 15 2017-08-28 15:28:13,058 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node2 2017-08-28 15:28:13,075 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,081 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node4 into one transmission 2017-08-28 15:28:13,081 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:13,082 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:13,083 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node4 2017-08-28 15:28:13,084 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 3 msgs to Node1 into one transmission 2017-08-28 15:28:13,084 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}', b'{"catchupTill":15,"seqNoStart":10,"ledgerId":1,"op":"CATCHUP_REQ","seqNoEnd":11}']) 2017-08-28 15:28:13,084 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"catchupTill\\":15,\\"seqNoStart\\":10,\\"ledgerId\\":1,\\"op\\":\\"CATCHUP_REQ\\",\\"seqNoEnd\\":11}"],"op":"BATCH"}' 2017-08-28 15:28:13,109 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"catchupTill\\":15,\\"seqNoStart\\":10,\\"ledgerId\\":1,\\"op\\":\\"CATCHUP_REQ\\",\\"seqNoEnd\\":11}"],"op":"BATCH"}' to Node1 2017-08-28 15:28:13,110 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"catchupTill":15,"seqNoStart":14,"ledgerId":1,"op":"CATCHUP_REQ","seqNoEnd":15}' to Node3 2017-08-28 15:28:13,117 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"catchupTill":15,"seqNoStart":14,"ledgerId":1,"op":"CATCHUP_REQ","seqNoEnd":15}' to Node3 2017-08-28 15:28:13,118 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"catchupTill":15,"seqNoStart":12,"ledgerId":1,"op":"CATCHUP_REQ","seqNoEnd":13}' to Node2 2017-08-28 15:28:13,118 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"catchupTill":15,"seqNoStart":12,"ledgerId":1,"op":"CATCHUP_REQ","seqNoEnd":13}' to Node2 2017-08-28 15:28:13,119 | DEBUG | keep_in_touch.py ( 68) | conns | Node6's connections changed from set() to {'Node1', 'Node3', 'Node2', 'Node4'} 2017-08-28 15:28:13,133 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node6 now connected to Node1 2017-08-28 15:28:13,135 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node6 now connected to Node3 2017-08-28 15:28:13,135 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node6 now connected to Node2 2017-08-28 15:28:13,139 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node6 now connected to Node4 2017-08-28 15:28:13,140 | DEBUG | motor.py ( 34) | set_status | Node6 changing status from starting to started_hungry 2017-08-28 15:28:13,140 | DEBUG | node.py ( 923) | checkInstances | Node6 choosing to start election on the basis of count 5 and nodes {'Node1', 'Node3', 'Node2', 'Node4'} 2017-08-28 15:28:13,147 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node6 has no ViewChangeDone message to send for view 2 2017-08-28 15:28:13,147 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node6 sending current state CURRENT_STATE{'viewNo': 2, 'primary': []} to lagged node Node1 2017-08-28 15:28:13,148 | DEBUG | node.py (2566) | send | Node6 sending message CURRENT_STATE{'viewNo': 2, 'primary': []} to 1 recipients: ['Node1'] 2017-08-28 15:28:13,148 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node6 has no ViewChangeDone message to send for view 2 2017-08-28 15:28:13,157 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node6 sending current state CURRENT_STATE{'viewNo': 2, 'primary': []} to lagged node Node3 2017-08-28 15:28:13,157 | DEBUG | node.py (2566) | send | Node6 sending message CURRENT_STATE{'viewNo': 2, 'primary': []} to 1 recipients: ['Node3'] 2017-08-28 15:28:13,158 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node6 has no ViewChangeDone message to send for view 2 2017-08-28 15:28:13,163 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node6 sending current state CURRENT_STATE{'viewNo': 2, 'primary': []} to lagged node Node2 2017-08-28 15:28:13,164 | DEBUG | node.py (2566) | send | Node6 sending message CURRENT_STATE{'viewNo': 2, 'primary': []} to 1 recipients: ['Node2'] 2017-08-28 15:28:13,164 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node6 has no ViewChangeDone message to send for view 2 2017-08-28 15:28:13,171 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node6 sending current state CURRENT_STATE{'viewNo': 2, 'primary': []} to lagged node Node4 2017-08-28 15:28:13,171 | DEBUG | node.py (2566) | send | Node6 sending message CURRENT_STATE{'viewNo': 2, 'primary': []} to 1 recipients: ['Node4'] 2017-08-28 15:28:13,180 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} to 1 recipients: ['Node1'] 2017-08-28 15:28:13,180 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node1'] 2017-08-28 15:28:13,188 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} to 1 recipients: ['Node3'] 2017-08-28 15:28:13,189 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node3'] 2017-08-28 15:28:13,189 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} to 1 recipients: ['Node2'] 2017-08-28 15:28:13,200 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node2'] 2017-08-28 15:28:13,204 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} to 1 recipients: ['Node4'] 2017-08-28 15:28:13,205 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'txnSeqNo': 9} to 1 recipients: ['Node4'] 2017-08-28 15:28:13,211 | DEBUG | looper.py ( 212) | runOnceNicely | it took 0.723 seconds to run once nicely 2017-08-28 15:28:13,220 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:28:13,231 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}']} 2017-08-28 15:28:13,238 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:28:13,251 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}', '{"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoEnd":15,"ledgerId":1,"ppSeqNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"viewNo":0,"op":"CONSISTENCY_PROOF","newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"}']} 2017-08-28 15:28:13,255 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,256 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node5') 2017-08-28 15:28:13,259 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,279 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,281 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node5') 2017-08-28 15:28:13,282 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,289 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CATCHUP_REP{'txns': {'15': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'txnTime': 1503933484, 'dest': '5qhSKMkpqui6taGLTecwqf', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, '14': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'txnTime': 1503933479, 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}}, 'ledgerId': 1, 'consProof': []} 2017-08-28 15:28:13,292 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'15': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'txnTime': 1503933484, 'dest': '5qhSKMkpqui6taGLTecwqf', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, '14': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'txnTime': 1503933479, 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}}, 'ledgerId': 1, 'op': 'CATCHUP_REP', 'consProof': []}, 'Node3') 2017-08-28 15:28:13,306 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'15': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'txnTime': 1503933484, 'dest': '5qhSKMkpqui6taGLTecwqf', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, '14': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'txnTime': 1503933479, 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}}, 'ledgerId': 1, 'consProof': []} 2017-08-28 15:28:13,308 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CATCHUP_REP{'txns': {'13': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'type': '1', 'role': '100', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'txnTime': 1503933471, 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, '12': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'type': '1', 'role': '2', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'txnTime': 1503933466, 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}}, 'ledgerId': 1, 'consProof': ['HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw', '6eiB4VZiNXZ35Z1SPBMg8HQ9QuBNLozoWqhLHcGgYAE4', 'AbFuZ9GiDG7iwYdCm8cC6UcKVLH4i1C3RH4jXb8msWs9', '4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,315 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'13': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'type': '1', 'role': '100', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'txnTime': 1503933471, 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, '12': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'type': '1', 'role': '2', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'txnTime': 1503933466, 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}}, 'ledgerId': 1, 'op': 'CATCHUP_REP', 'consProof': ['HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw', '6eiB4VZiNXZ35Z1SPBMg8HQ9QuBNLozoWqhLHcGgYAE4', 'AbFuZ9GiDG7iwYdCm8cC6UcKVLH4i1C3RH4jXb8msWs9', '4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']}, 'Node2') 2017-08-28 15:28:13,316 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'13': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'type': '1', 'role': '100', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'txnTime': 1503933471, 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, '12': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'type': '1', 'role': '2', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'txnTime': 1503933466, 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}}, 'ledgerId': 1, 'consProof': ['HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw', '6eiB4VZiNXZ35Z1SPBMg8HQ9QuBNLozoWqhLHcGgYAE4', 'AbFuZ9GiDG7iwYdCm8cC6UcKVLH4i1C3RH4jXb8msWs9', '4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,320 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: BATCH{'signature': None, 'messages': ['{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}']} 2017-08-28 15:28:13,321 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:28:13,340 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}', '{"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"CONSISTENCY_PROOF","seqNoStart":9,"ledgerId":1,"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","seqNoEnd":15,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":0,"ppSeqNo":6}']} 2017-08-28 15:28:13,342 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,346 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node4') 2017-08-28 15:28:13,364 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,365 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,370 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node4') 2017-08-28 15:28:13,373 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,374 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node5 2017-08-28 15:28:13,374 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,375 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 0, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node5 2017-08-28 15:28:13,376 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,376 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node3: CATCHUP_REP{'txns': {'15': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'txnTime': 1503933484, 'dest': '5qhSKMkpqui6taGLTecwqf', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, '14': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'txnTime': 1503933479, 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}}, 'ledgerId': 1, 'consProof': []} 2017-08-28 15:28:13,377 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 2 transactions in the catchup from Node3 2017-08-28 15:28:13,377 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:13,400 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 2 of them now, from 14 to 15 2017-08-28 15:28:13,401 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 0 catchup replies with sequence numbers [] 2017-08-28 15:28:13,401 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node2: CATCHUP_REP{'txns': {'13': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'type': '1', 'role': '100', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'txnTime': 1503933471, 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, '12': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'type': '1', 'role': '2', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'txnTime': 1503933466, 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}}, 'ledgerId': 1, 'consProof': ['HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw', '6eiB4VZiNXZ35Z1SPBMg8HQ9QuBNLozoWqhLHcGgYAE4', 'AbFuZ9GiDG7iwYdCm8cC6UcKVLH4i1C3RH4jXb8msWs9', '4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,413 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 2 transactions in the catchup from Node2 2017-08-28 15:28:13,413 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:13,422 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 4 of them now, from 12 to 15 2017-08-28 15:28:13,423 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 0 catchup replies with sequence numbers [] 2017-08-28 15:28:13,426 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node4 2017-08-28 15:28:13,427 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,429 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node4 2017-08-28 15:28:13,436 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,455 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node6's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:13,456 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node6 needs 1 ViewChangeDone messages 2017-08-28 15:28:13,456 | DEBUG | primary_selector.py ( 265) | _startSelection | Node6 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:28:13,458 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node6's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:13,462 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node6 got view change quorum (2 >= 2) 2017-08-28 15:28:13,462 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node6 received ViewChangeDone from primary Node3 2017-08-28 15:28:13,463 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node6 found acceptable primary Node3 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:28:13,482 | DEBUG | primary_selector.py ( 270) | _startSelection | Node6 cannot start primary selection since mode is 400 2017-08-28 15:28:13,486 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node6's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:13,487 | DEBUG | primary_selector.py ( 270) | _startSelection | Node6 cannot start primary selection since mode is 400 2017-08-28 15:28:13,488 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node6's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:13,491 | DEBUG | primary_selector.py ( 270) | _startSelection | Node6 cannot start primary selection since mode is 400 2017-08-28 15:28:13,506 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 3 msgs to Node4 into one transmission 2017-08-28 15:28:13,507 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"viewNo":2,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:13,510 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:13,511 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node4 2017-08-28 15:28:13,515 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 3 msgs to Node1 into one transmission 2017-08-28 15:28:13,516 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"viewNo":2,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:13,516 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:13,519 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node1 2017-08-28 15:28:13,534 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 3 msgs to Node3 into one transmission 2017-08-28 15:28:13,534 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"viewNo":2,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:13,535 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node3: b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:13,543 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node3 2017-08-28 15:28:13,548 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 3 msgs to Node2 into one transmission 2017-08-28 15:28:13,549 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"viewNo":2,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:13,549 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node2: b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:13,558 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node2 2017-08-28 15:28:13,576 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:13,577 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"consProof":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CATCHUP_REP","txns":{"10":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","txnTime":1503933200,"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe61","type":"1","reqId":1503933200605368},"11":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","txnTime":1503933459,"verkey":"~RmCt3RtDvKDfQBSKxo4qvy","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","type":"1","role":"2","reqId":1503933459424488}},"ledgerId":1}']} 2017-08-28 15:28:13,591 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"consProof":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CATCHUP_REP","txns":{"10":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","txnTime":1503933200,"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe61","type":"1","reqId":1503933200605368},"11":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","txnTime":1503933459,"verkey":"~RmCt3RtDvKDfQBSKxo4qvy","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","type":"1","role":"2","reqId":1503933459424488}},"ledgerId":1}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:28:13,599 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"newMerkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"CONSISTENCY_PROOF","seqNoEnd":15,"ledgerId":1,"ppSeqNo":6,"seqNoStart":9,"viewNo":0,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC","FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"]}', '{"consProof":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CATCHUP_REP","txns":{"10":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","txnTime":1503933200,"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe61","type":"1","reqId":1503933200605368},"11":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","txnTime":1503933459,"verkey":"~RmCt3RtDvKDfQBSKxo4qvy","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","type":"1","role":"2","reqId":1503933459424488}},"ledgerId":1}']} 2017-08-28 15:28:13,604 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,605 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node1') 2017-08-28 15:28:13,611 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,623 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,631 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node1') 2017-08-28 15:28:13,639 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,640 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CATCHUP_REP{'txns': {'11': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'txnTime': 1503933459, 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, '10': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'txnTime': 1503933200, 'dest': 'V4SGRU86Z58d6TV7PBUe61', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1'}}, 'ledgerId': 1, 'consProof': ['8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY', '5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna', 'A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,649 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'txns': {'11': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'txnTime': 1503933459, 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, '10': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'txnTime': 1503933200, 'dest': 'V4SGRU86Z58d6TV7PBUe61', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1'}}, 'ledgerId': 1, 'op': 'CATCHUP_REP', 'consProof': ['8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY', '5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna', 'A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']}, 'Node1') 2017-08-28 15:28:13,650 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CATCHUP_REP{'txns': {'11': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'txnTime': 1503933459, 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, '10': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'txnTime': 1503933200, 'dest': 'V4SGRU86Z58d6TV7PBUe61', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1'}}, 'ledgerId': 1, 'consProof': ['8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY', '5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna', 'A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,660 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node1 2017-08-28 15:28:13,675 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,676 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node1 2017-08-28 15:28:13,682 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node6 cannot process consistency proof since ledger state is 2 2017-08-28 15:28:13,683 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node6 received catchup reply from Node1: CATCHUP_REP{'txns': {'11': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'txnTime': 1503933459, 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, '10': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'txnTime': 1503933200, 'dest': 'V4SGRU86Z58d6TV7PBUe61', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1'}}, 'ledgerId': 1, 'consProof': ['8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY', '5BCQoW4vvq573NaYeKU6uTgZVrdKxzuz7J1QdwN5Njna', 'A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz']} 2017-08-28 15:28:13,694 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node6 found 2 transactions in the catchup from Node1 2017-08-28 15:28:13,694 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node6 merging all received catchups 2017-08-28 15:28:13,694 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node6 merged catchups, there are 6 of them now, from 10 to 15 2017-08-28 15:28:13,707 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 11, 15, b"\x1e\t\xbc02\xb7\xf2'\xd8Z+\x97.\xf9u\xa0D\xb1o\x8dS\xc1\x9fo\xb3\x92(Y\x89\x87\x83\xe6", b'\xcc\xe8+\x8d\xa3R\xe0\x8fX\xdf\xa2\x91\x9ea\x9b\x10c\x12\xa8]+\xdbD~\xa7U\xf6]>\xce_F', [b'j\xd3\x18\xdc\x8f\x1e\xb1\xc7\xc7\x14\xd1\xc1\xf8/i\xd3\xad\xb2\x94)8`\xff\x95\xc2^Ct\x15{\x10\x85', b'>\x0bu!\xe0\xdaw[\x1b5a\x8a*y\x9fj\xe4R\xa0f\xe2\x8eN4C\x9d\xca\xa0_\x1a\xf8c', b'\x87y\xf2\x1e|:\xe7\x17\xf3\xd3\xc7;o8\xce\x05u\x01Qn\n\x95\xc4\xee\xd8\xc00\xf6\x9b\xfb\xc7\xe2', b"\xe4\x84|W\xe7\xa2\x88\xb8p\r\xe7\xf6/3\xfc\xc4\x1a\x11\xea\xce\x90dL\xc2m\xb9\xe4!\x17F\xc5'", b'\xd7\r\x82\x86t\xa38`\xaa\xbb0\xa9J\xccM\xd1\xbci\xb8_~^\x18\x1a\x92\x934\xcd\xcaB\x1f\xb9'] 2017-08-28 15:28:13,711 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 13, 15, b'\xac\x8c\xc5\xa0\x8cw\x03\x88.P\x13]\xabb\xe9\x9d\x14X\x1196\xd27\x97DY\xdcZk\x8f\x08E', b'\xcc\xe8+\x8d\xa3R\xe0\x8fX\xdf\xa2\x91\x9ea\x9b\x10c\x12\xa8]+\xdbD~\xa7U\xf6]>\xce_F', [b'\xfah\xba\xcd\xa2P\xe0z\x03\xc7\x93\xc3B-_1<%\xa9\xfc`\xfb\x8e\rv\xc4,\x10\\3=\xaa', b'S\xf3z\xc3\x02\x9b\xfb\x14\xb8J\xf8\xbaHB\xebc5R\x98\xd9i\xe6\xe1G\xaa\x86u\x10\x04\t\xf3\xa1', b'\x8e\x7f\xc8\xc3\x92\\v\t\xff\xff\xc6\xf1\xd2O/\xedN][\xd2R\xff\xb2\x81\x00\xb5\x04N\xe2w\x01\xe0', b'/\xc4\xc5\xc2^\xc3\x85V\xb9\x83\x1a\x90\xcc\x18\xf1\xfa\xeeO\xcf\xbbD\x95 \xfa\xe8\xdb\xe6x\xa2\x8fK\xc4', b'\xd7\r\x82\x86t\xa38`\xaa\xbb0\xa9J\xccM\xd1\xbci\xb8_~^\x18\x1a\x92\x934\xcd\xcaB\x1f\xb9'] 2017-08-28 15:28:13,728 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node6 verifying proof for 15, 15, b'\xcc\xe8+\x8d\xa3R\xe0\x8fX\xdf\xa2\x91\x9ea\x9b\x10c\x12\xa8]+\xdbD~\xa7U\xf6]>\xce_F', b'\xcc\xe8+\x8d\xa3R\xe0\x8fX\xdf\xa2\x91\x9ea\x9b\x10c\x12\xa8]+\xdbD~\xa7U\xf6]>\xce_F', [] 2017-08-28 15:28:13,754 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node6 processed 6 catchup replies with sequence numbers [10, 11, 12, 13, 14, 15] 2017-08-28 15:28:13,771 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node6 completed catching up ledger 1, caught up 6 in total 2017-08-28 15:28:13,775 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node6 caught up to 9 txns in the last catchup 2017-08-28 15:28:13,776 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:0 set last ordered as (2, 6) 2017-08-28 15:28:13,776 | DEBUG | replica.py (2174) | _remove_till_caught_up_3pc | Node6:0 going to remove messages for 0 3PC keys 2017-08-28 15:28:13,778 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node6:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:28:13,779 | INFO | node.py (1507) | allLedgersCaughtUp | CATCH-UP: Node6 caught up till (2, 6) 2017-08-28 15:28:13,779 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node6 processed 0 stashed ordered requests 2017-08-28 15:28:13,780 | DEBUG | monitor.py ( 192) | reset | Node6's Monitor being reset 2017-08-28 15:28:13,784 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node6 got view change quorum (4 >= 2) 2017-08-28 15:28:13,798 | DEBUG | node.py (1531) | is_catchup_needed | Node6 is caught up for the current view 2 2017-08-28 15:28:13,798 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node6 does not need any more catchups 2017-08-28 15:28:13,805 | DEBUG | primary_decider.py ( 131) | send | Node6's elector sending VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [(0, 7, '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node3'} 2017-08-28 15:28:13,805 | DEBUG | primary_selector.py ( 280) | _startSelection | Node6 starting selection 2017-08-28 15:28:13,808 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node6:0 selected primary Node3:0 for instance 0 (view 2) 2017-08-28 15:28:13,809 | INFO | node.py ( 503) | start_participating | Node6 started participating 2017-08-28 15:28:13,809 | DEBUG | replica.py ( 406) | primaryName | Node6:0 setting primaryName for view no 2 to: Node3:0 2017-08-28 15:28:13,823 | DEBUG | replica.py (1678) | _gc | Node6:0 cleaning up till (2, 6) 2017-08-28 15:28:13,824 | DEBUG | replica.py (1693) | _gc | Node6:0 found 0 3-phase keys to clean 2017-08-28 15:28:13,824 | DEBUG | replica.py (1695) | _gc | Node6:0 found 0 request keys to clean 2017-08-28 15:28:13,824 | DEBUG | replica.py ( 311) | h | Node6:0 set watermarks as 0 300 2017-08-28 15:28:13,836 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node6:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 7, '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:28:13,836 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node6:1 selected primary Node4:1 for instance 1 (view 2) 2017-08-28 15:28:13,842 | DEBUG | replica.py ( 406) | primaryName | Node6:1 setting primaryName for view no 2 to: Node4:1 2017-08-28 15:28:13,843 | DEBUG | replica.py (1678) | _gc | Node6:1 cleaning up till (0, 0) 2017-08-28 15:28:13,847 | DEBUG | replica.py (1693) | _gc | Node6:1 found 0 3-phase keys to clean 2017-08-28 15:28:13,847 | DEBUG | replica.py (1695) | _gc | Node6:1 found 0 request keys to clean 2017-08-28 15:28:13,848 | DEBUG | replica.py ( 311) | h | Node6:1 set watermarks as 0 300 2017-08-28 15:28:13,857 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (0, 0) 2017-08-28 15:28:13,858 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:28:13,858 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node6:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 7, '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:28:13,866 | DEBUG | node.py (2566) | send | Node6 sending message VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [(0, 7, '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node3'} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:28:13,870 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node5 2017-08-28 15:28:13,870 | DEBUG | zstack.py ( 726) | transmit | Remote Node5 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:28:13,870 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node5 2017-08-28 15:28:13,887 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node4 2017-08-28 15:28:13,892 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node4 2017-08-28 15:28:13,892 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node1 2017-08-28 15:28:13,892 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node1 2017-08-28 15:28:13,893 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node3 2017-08-28 15:28:13,893 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node3 2017-08-28 15:28:13,904 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node2 2017-08-28 15:28:13,905 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":2,"ledgerInfo":[[0,7,"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","op":"VIEW_CHANGE_DONE"}' to Node2 2017-08-28 15:28:13,907 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:28:13,924 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,936 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node1') 2017-08-28 15:28:13,936 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,947 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,949 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node4') 2017-08-28 15:28:13,949 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,959 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,974 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node3') 2017-08-28 15:28:13,975 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,981 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,984 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF'}, 'Node2') 2017-08-28 15:28:13,989 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} 2017-08-28 15:28:13,996 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node1 2017-08-28 15:28:13,997 | DEBUG | ledger_manager.py ( 351) | canProcessConsistencyProof | Node6 cannot process consistency proof since canSync is False 2017-08-28 15:28:14,005 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node4 2017-08-28 15:28:14,011 | DEBUG | ledger_manager.py ( 351) | canProcessConsistencyProof | Node6 cannot process consistency proof since canSync is False 2017-08-28 15:28:14,012 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node3 2017-08-28 15:28:14,013 | DEBUG | ledger_manager.py ( 351) | canProcessConsistencyProof | Node6 cannot process consistency proof since canSync is False 2017-08-28 15:28:14,013 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node6 received consistency proof: CONSISTENCY_PROOF{'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', 'dkwHWCFiwe2wbhEexYxaQWdtpxCuP3hfXcrEQS5DZFC', 'FnKGRvRo7xL57adPnN2aqCYoUpTEnoD9oTgy3BKedf73', 'GP38KJDvXgfPRRZYoabDS3yMBEprbtFzBWdPehWc7TSN', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9, 'seqNoEnd': 15, 'viewNo': 0, 'newMerkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1} from Node2 2017-08-28 15:28:14,036 | DEBUG | ledger_manager.py ( 351) | canProcessConsistencyProof | Node6 cannot process consistency proof since canSync is False 2017-08-28 15:28:14,075 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.713 seconds 2017-08-28 15:28:18,518 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 80 2017-08-28 15:28:18,519 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:18,519 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:28:18,531 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:28:18,532 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:28:18,532 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:28:18,532 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:28:18,535 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-08-28 15:28:25,211 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:25,224 | DEBUG | zstack.py ( 661) | handlePingPong | Node6 got ping from Node5 2017-08-28 15:28:25,224 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 ponged Node5 2017-08-28 15:28:25,239 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'po' to Node5 2017-08-28 15:28:25,252 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'po' to Node5 2017-08-28 15:28:25,377 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:25,378 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"op":"CURRENT_STATE"}', '{"ledgerId":0,"viewNo":2,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"ppSeqNo":6}', '{"ledgerId":1,"viewNo":null,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"ppSeqNo":null}', '{"ledgerId":2,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"ppSeqNo":null}']} 2017-08-28 15:28:25,388 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"op":"CURRENT_STATE"}', '{"ledgerId":0,"viewNo":2,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"ppSeqNo":6}', '{"ledgerId":1,"viewNo":null,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"ppSeqNo":null}', '{"ledgerId":2,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"ppSeqNo":null}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:28:25,403 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"op":"CURRENT_STATE"}', '{"ledgerId":0,"viewNo":2,"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","op":"LEDGER_STATUS","txnSeqNo":7,"ppSeqNo":6}', '{"ledgerId":1,"viewNo":null,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","op":"LEDGER_STATUS","txnSeqNo":15,"ppSeqNo":null}', '{"ledgerId":2,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"ppSeqNo":null}']} 2017-08-28 15:28:25,408 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:25,408 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}], 'op': 'CURRENT_STATE'}, 'Node5') 2017-08-28 15:28:25,409 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} 2017-08-28 15:28:25,409 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:25,410 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'txnSeqNo': 7, 'viewNo': 2, 'ppSeqNo': 6, 'ledgerId': 0, 'op': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:25,420 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} 2017-08-28 15:28:25,428 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:25,428 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1, 'op': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:25,428 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} 2017-08-28 15:28:25,434 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:25,434 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'op': 'LEDGER_STATUS'}, 'Node5') 2017-08-28 15:28:25,444 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} 2017-08-28 15:28:25,445 | DEBUG | node.py ( 889) | process_current_state_message | Node6 processing current state CURRENT_STATE{'viewNo': 2, 'primary': [{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}]} from Node5 2017-08-28 15:28:25,463 | DEBUG | node.py (1187) | sendToElector | Node6 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'}, 'Node5') 2017-08-28 15:28:25,465 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': 2, 'ppSeqNo': 6, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} from Node5 2017-08-28 15:28:25,465 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 7 with 7 2017-08-28 15:28:25,466 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 0 of size 7 with 7 2017-08-28 15:28:25,466 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} from Node5 2017-08-28 15:28:25,479 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 15 with 15 2017-08-28 15:28:25,479 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 1 of size 15 with 15 2017-08-28 15:28:25,480 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node6 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} from Node5 2017-08-28 15:28:25,486 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:25,503 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node6 comparing its ledger 2 of size 0 with 0 2017-08-28 15:28:25,506 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node6 found out from {'Node5', 'Node4', 'Node3', 'Node1', 'Node2'} that its ledger of type 2 is latest 2017-08-28 15:28:25,514 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node6's primary selector started processing of ViewChangeDone msg from Node5 : VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} 2017-08-28 15:28:25,524 | DEBUG | message_processor.py ( 29) | discard | Node6 discarding message VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3'} because it already decided primary which is Node3:0 2017-08-28 15:28:25,553 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.178 seconds 2017-08-28 15:28:26,625 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action partial(request_txns_if_needed) with id 2 2017-08-28 15:28:26,626 | DEBUG | ledger_manager.py ( 136) | request_txns_if_needed | Node6 not missing any transactions for ledger 0 2017-08-28 15:28:26,652 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:26,675 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:28:26,676 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:26,676 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:26,676 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:26,677 | DEBUG | zstack.py ( 643) | sendPingPong | Node6 pinged Node5 2017-08-28 15:28:26,679 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:26,721 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'pi' to Node5 2017-08-28 15:28:26,721 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'pi' to Node5 2017-08-28 15:28:26,771 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:28:26,776 | DEBUG | zstack.py ( 667) | handlePingPong | Node6 got pong from Node5 2017-08-28 15:28:26,800 | DEBUG | keep_in_touch.py ( 68) | conns | Node6's connections changed from {'Node1', 'Node3', 'Node2', 'Node4'} to {'Node1', 'Node3', 'Node2', 'Node4', 'Node5'} 2017-08-28 15:28:26,801 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node6 now connected to Node5 2017-08-28 15:28:26,803 | DEBUG | motor.py ( 34) | set_status | Node6 changing status from started_hungry to started 2017-08-28 15:28:26,804 | DEBUG | node.py ( 923) | checkInstances | Node6 choosing to start election on the basis of count 6 and nodes {'Node1', 'Node3', 'Node2', 'Node4', 'Node5'} 2017-08-28 15:28:26,805 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node6 sending current state CURRENT_STATE{'viewNo': 2, 'primary': [VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node3'}]} to lagged node Node5 2017-08-28 15:28:26,809 | DEBUG | node.py (2566) | send | Node6 sending message CURRENT_STATE{'viewNo': 2, 'primary': [VIEW_CHANGE_DONE{'viewNo': 2, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node3'}]} to 1 recipients: ['Node5'] 2017-08-28 15:28:26,810 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ledgerId': 0, 'txnSeqNo': 7} to 1 recipients: ['Node5'] 2017-08-28 15:28:26,828 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'txnSeqNo': 15} to 1 recipients: ['Node5'] 2017-08-28 15:28:26,828 | DEBUG | node.py (2566) | send | Node6 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0} to 1 recipients: ['Node5'] 2017-08-28 15:28:26,839 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 4 msgs to Node5 into one transmission 2017-08-28 15:28:26,848 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"viewNo":2,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node3","viewNo":2}],"op":"CURRENT_STATE"}', b'{"merkleRoot":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","txnSeqNo":7,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","txnSeqNo":15,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS"}', b'{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"ppSeqNo":null,"ledgerId":2,"op":"LEDGER_STATUS"}']) 2017-08-28 15:28:26,848 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,15,\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node3\\",\\"viewNo\\":2}],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"txnSeqNo\\":15,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' 2017-08-28 15:28:26,871 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"viewNo\\":2,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,15,\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node3\\",\\"viewNo\\":2}],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ\\",\\"txnSeqNo\\":7,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"txnSeqNo\\":15,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\"}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\"}"],"op":"BATCH"}' to Node5 2017-08-28 15:28:26,873 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.103 seconds 2017-08-28 15:28:28,608 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 82 2017-08-28 15:28:28,616 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:28,616 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:28:28,616 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:28:28,634 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:28:28,634 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:28:28,635 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:28:28,635 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-08-28 15:28:28,643 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action partial(request_txns_if_needed) with id 4 2017-08-28 15:28:28,644 | DEBUG | ledger_manager.py ( 136) | request_txns_if_needed | Node6 not missing any transactions for ledger 1 2017-08-28 15:28:28,719 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.130 seconds 2017-08-28 15:28:31,254 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action partial(request_CPs_if_needed) with id 1 2017-08-28 15:28:37,908 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action partial(request_CPs_if_needed) with id 3 2017-08-28 15:28:38,673 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 83 2017-08-28 15:28:38,673 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:38,674 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:28:38,674 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:28:38,682 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:28:38,682 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:28:38,695 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:28:38,695 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-08-28 15:28:41,690 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:41,691 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:28:41,692 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:41,692 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:41,692 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:41,694 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:48,700 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 84 2017-08-28 15:28:48,700 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:48,701 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:28:48,715 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:28:48,716 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:28:48,716 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:28:48,718 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:28:48,728 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 85 to run in 10 seconds 2017-08-28 15:28:56,708 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:28:56,708 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:28:56,709 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:28:56,709 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:28:56,709 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:28:56,710 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:28:58,731 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 85 2017-08-28 15:28:58,747 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:28:58,748 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:28:58,749 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:28:58,749 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:28:58,750 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:28:58,767 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:28:58,768 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-08-28 15:29:07,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 79 2017-08-28 15:29:07,027 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:29:07,027 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:07,030 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 87 to run in 60 seconds 2017-08-28 15:29:07,031 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 12 2017-08-28 15:29:07,035 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:29:07,036 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-08-28 15:29:08,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 86 2017-08-28 15:29:08,818 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:08,819 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:08,822 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:08,823 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:08,823 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:08,827 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:08,827 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-08-28 15:29:10,982 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action _check_view_change_completed with id 81 2017-08-28 15:29:10,987 | DEBUG | node.py ( 993) | _check_view_change_completed | Node6 running the scheduled check for view change completion 2017-08-28 15:29:10,995 | DEBUG | node.py ( 995) | _check_view_change_completed | Node6 already completion view change 2017-08-28 15:29:11,733 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:29:11,734 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:29:11,734 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:29:11,737 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:29:11,748 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:29:11,749 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:29:18,829 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 88 2017-08-28 15:29:18,830 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:18,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:18,831 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:18,835 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:18,839 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:18,840 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:18,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-08-28 15:29:26,757 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:29:26,757 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:29:26,758 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:29:26,758 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:29:26,760 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:29:26,772 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:29:28,853 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 89 2017-08-28 15:29:28,854 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:28,854 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:28,856 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:28,856 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:28,856 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:28,859 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:28,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-08-28 15:29:38,872 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 90 2017-08-28 15:29:38,875 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:38,878 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:38,882 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:38,882 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:38,882 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:38,883 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:38,883 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-08-28 15:29:41,762 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:29:41,773 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:29:41,773 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:29:41,773 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:29:41,773 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:29:41,774 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:29:48,909 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 91 2017-08-28 15:29:48,914 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:48,923 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:48,927 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:48,928 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:48,928 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:48,928 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:48,929 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-08-28 15:29:56,764 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:29:56,788 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:29:56,788 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:29:56,788 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:29:56,789 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:29:56,789 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:29:58,938 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 92 2017-08-28 15:29:58,938 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:29:58,939 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:29:58,947 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:29:58,952 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:29:58,955 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:29:58,956 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:29:58,956 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 93 to run in 10 seconds 2017-08-28 15:30:07,047 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 87 2017-08-28 15:30:07,050 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:30:07,054 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:07,054 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 94 to run in 60 seconds 2017-08-28 15:30:07,054 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 13 2017-08-28 15:30:07,054 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:30:07,055 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-08-28 15:30:08,977 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 93 2017-08-28 15:30:08,978 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:08,978 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:08,978 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:08,995 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:08,996 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:08,996 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:08,996 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-08-28 15:30:11,797 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:30:11,797 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:30:11,798 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:30:11,798 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:30:11,801 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:30:11,802 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:30:18,999 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 95 2017-08-28 15:30:19,000 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:19,003 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:19,003 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:19,010 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:19,011 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:19,014 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:19,015 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-08-28 15:30:26,811 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:30:26,828 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:30:26,828 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:30:26,828 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:30:26,829 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:30:26,832 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:30:29,041 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 96 2017-08-28 15:30:29,042 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:29,042 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:29,042 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:29,042 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:29,042 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:29,043 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:29,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-08-28 15:30:39,065 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 97 2017-08-28 15:30:39,066 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:39,067 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:39,067 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:39,088 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:39,089 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:39,089 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:39,089 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-08-28 15:30:41,828 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:30:41,832 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:30:41,833 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:30:41,851 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:30:41,852 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:30:41,852 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:30:49,110 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 98 2017-08-28 15:30:49,121 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:49,124 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:49,124 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:49,124 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:49,128 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:49,130 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:49,130 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 99 to run in 10 seconds 2017-08-28 15:30:56,842 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:30:56,845 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:30:56,846 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:30:56,848 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:30:56,849 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:30:56,849 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:30:59,167 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 99 2017-08-28 15:30:59,168 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:30:59,168 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:30:59,168 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:30:59,169 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:30:59,171 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:30:59,173 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:30:59,183 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 100 to run in 10 seconds 2017-08-28 15:31:07,080 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 94 2017-08-28 15:31:07,086 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:31:07,086 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:07,087 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 101 to run in 60 seconds 2017-08-28 15:31:07,089 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 14 2017-08-28 15:31:07,090 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:31:07,090 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 15 to run in 60 seconds 2017-08-28 15:31:09,190 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 100 2017-08-28 15:31:09,193 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:09,195 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:09,196 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:09,196 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:09,196 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:09,197 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:09,200 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 102 to run in 10 seconds 2017-08-28 15:31:11,849 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:31:11,850 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:31:11,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:31:11,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:31:11,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:31:11,856 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:31:16,159 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.125 seconds 2017-08-28 15:31:19,236 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 102 2017-08-28 15:31:19,237 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:19,239 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:19,244 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:19,244 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:19,245 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:19,245 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:19,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 103 to run in 10 seconds 2017-08-28 15:31:26,883 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:31:26,902 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:31:26,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:31:26,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:31:26,909 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:31:26,910 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:31:29,261 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 103 2017-08-28 15:31:29,267 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:29,267 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:29,270 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:29,271 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:29,273 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:29,273 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:29,274 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 104 to run in 10 seconds 2017-08-28 15:31:39,276 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 104 2017-08-28 15:31:39,279 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:39,280 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:39,287 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:39,288 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:39,288 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:39,288 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:39,288 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 105 to run in 10 seconds 2017-08-28 15:31:41,899 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:31:41,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:31:41,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:31:41,901 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:31:41,902 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:31:41,904 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:31:49,290 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 105 2017-08-28 15:31:49,296 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:49,297 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:49,297 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:49,316 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:49,317 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:49,317 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:49,317 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 106 to run in 10 seconds 2017-08-28 15:31:56,913 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:31:56,914 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:31:56,914 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:31:56,915 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:31:56,915 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:31:56,923 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:31:59,342 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 106 2017-08-28 15:31:59,351 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:31:59,351 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:31:59,352 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:31:59,367 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:31:59,367 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:31:59,368 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:31:59,368 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 107 to run in 10 seconds 2017-08-28 15:32:07,110 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 101 2017-08-28 15:32:07,111 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:32:07,114 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:07,114 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 108 to run in 60 seconds 2017-08-28 15:32:07,115 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 15 2017-08-28 15:32:07,115 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:32:07,115 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 16 to run in 60 seconds 2017-08-28 15:32:09,384 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 107 2017-08-28 15:32:09,384 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:09,385 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:09,393 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:09,393 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:09,394 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:32:09,394 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:09,394 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 109 to run in 10 seconds 2017-08-28 15:32:11,927 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:32:11,940 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:32:11,940 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:32:11,940 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:32:11,941 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:32:11,941 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:32:19,430 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 109 2017-08-28 15:32:19,435 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:19,435 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:19,436 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:19,436 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:19,438 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node6 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:32:19,438 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:19,448 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 110 to run in 10 seconds 2017-08-28 15:32:21,297 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:32:21,323 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1503934341241353 2017-08-28 15:32:21,323 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,342 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,348 | DEBUG | node.py (1690) | processRequest | Node6 received client request: SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:32:21,357 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,358 | DEBUG | propagator.py ( 148) | propagate | Node6 propagating XhYtvJqezMUKfF6KVNaGmT request 1503934341241353 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:32:21,362 | DEBUG | node.py (2566) | send | Node6 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:21,363 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since not finalised 2017-08-28 15:32:21,381 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934341241353,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:32:21,384 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:32:21,397 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:32:21,401 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:32:21,410 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:32:21,419 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:32:21,426 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:32:21,427 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:32:21,436 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:32:21,451 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:32:21,457 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk","reqId":1503934341241353,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.7","node_port":9701,"client_ip":"10.0.0.7","alias":"Node6","services":[]},"dest":"6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:32:21,478 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:32:21,493 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1503934341241353 2017-08-28 15:32:21,493 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,496 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'op': 'PROPAGATE'}, 'Node5') 2017-08-28 15:32:21,496 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,505 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1503934341241353 2017-08-28 15:32:21,521 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,522 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'op': 'PROPAGATE'}, 'Node3') 2017-08-28 15:32:21,522 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,531 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1503934341241353 2017-08-28 15:32:21,538 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,538 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'op': 'PROPAGATE'}, 'Node4') 2017-08-28 15:32:21,549 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,551 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1503934341241353 2017-08-28 15:32:21,562 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,563 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'op': 'PROPAGATE'}, 'Node1') 2017-08-28 15:32:21,572 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,573 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1503934341241353 2017-08-28 15:32:21,584 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,585 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'op': 'PROPAGATE'}, 'Node2') 2017-08-28 15:32:21,585 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,597 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,609 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,617 | DEBUG | propagator.py ( 213) | forward | Node6 forwarding request ('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353) to 0 replicas 2017-08-28 15:32:21,618 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,624 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,632 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since already forwarded 2017-08-28 15:32:21,645 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,650 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,655 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'node_port': 9701, 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since already forwarded 2017-08-28 15:32:21,661 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,662 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,672 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node6', 'services': [], 'client_ip': '10.0.0.7', 'node_ip': '10.0.0.7', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since already forwarded 2017-08-28 15:32:21,681 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}} 2017-08-28 15:32:21,687 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-08-28 15:32:21,690 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'services': [], 'client_ip': '10.0.0.7', 'alias': 'Node6', 'node_port': 9701}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since already forwarded 2017-08-28 15:32:21,765 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.496 seconds 2017-08-28 15:32:21,811 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:21,821 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'ppTime': 1503934341, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:21,827 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'discarded': 1, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'ppSeqNo': 7, 'instId': 1, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'stateRootHash': None, 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node4') 2017-08-28 15:32:21,828 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'ppTime': 1503934341, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:21,841 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:1 received PRE-PREPARE(2, 7) from Node4:1 at 250637.990665112 2017-08-28 15:32:21,856 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:21,856 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:21,856 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:32:21,865 | DEBUG | replica.py (1822) | enqueue_pre_prepare | Queueing pre-prepares due to unavailability of previous pre-prepares. PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353)], 'ppTime': 1503934341, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} from Node4:1 2017-08-28 15:32:21,869 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:21,870 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'ppTime': 1503934341, 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:21,882 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'instId': 0, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'discarded': 0, 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'viewNo': 2, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node3') 2017-08-28 15:32:21,883 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1503934341241353]], 'ppTime': 1503934341, 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:21,906 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:0 received PRE-PREPARE(2, 7) from Node3:0 at 250638.054801153 2017-08-28 15:32:21,913 | DEBUG | replica.py (1034) | validate_pre_prepare | Node6:0 state root before processing PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353)], 'ppTime': 1503934341, 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} is b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9', 67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ 2017-08-28 15:32:21,926 | WARNING | replica.py ( 655) | processReqDuringBatch | Node6:0 encountered exception UnauthorizedClientRequest('XhYtvJqezMUKfF6KVNaGmT is not a steward of node 6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G',) while processing Request: {'signature': '5r2UjHCY4xdr1gZMc6RbtWBPX28UpwRj4CgdHkkiABEaS5MqrSjswS3VUssmGEBTzpJ3cnKjmuqqJ2dakXecWNKk', 'reqId': 1503934341241353, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.7', 'node_port': 9701, 'client_ip': '10.0.0.7', 'alias': 'Node6', 'services': []}, 'dest': '6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, will reject 2017-08-28 15:32:21,927 | DEBUG | replica.py ( 934) | doPrepare | Node6:0 Sending PREPARE(2, 7) at 250638.075941849 2017-08-28 15:32:21,928 | DEBUG | replica.py (2133) | send | Node6:0 sending Prepare 2017-08-28 15:32:21,941 | TRACE | replica.py (2134) | send | Node6:0 sending PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:21,941 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:21,941 | DEBUG | replica.py ( 612) | trackBatches | Node6:0 tracking batch for PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353)], 'ppTime': 1503934341, 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} with state root b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9' 2017-08-28 15:32:21,946 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:0 processed incoming PRE-PREPARE(2, 7) 2017-08-28 15:32:21,947 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:21,947 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:21,955 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:32:21,985 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934341241353,"reason":"client request invalid: UnauthorizedClientRequest(\'XhYtvJqezMUKfF6KVNaGmT is not a steward of node 6G9QhQa3HWjRKeRmEvEkLbWWf2t7cw6KLtafzi494G4G\',)","op":"REJECT","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:32:21,993 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:21,994 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:21,994 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:21,994 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:32:22,006 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:32:22,016 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,027 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 7, 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node4') 2017-08-28 15:32:22,028 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,028 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,033 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'viewNo': 2, 'ppSeqNo': 7, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node3') 2017-08-28 15:32:22,034 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,034 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,041 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppSeqNo': 7, 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:32:22,041 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,065 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node5 2017-08-28 15:32:22,066 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node5 2017-08-28 15:32:22,075 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node4 2017-08-28 15:32:22,076 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node4 2017-08-28 15:32:22,076 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node1 2017-08-28 15:32:22,093 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node1 2017-08-28 15:32:22,094 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node3 2017-08-28 15:32:22,108 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node3 2017-08-28 15:32:22,108 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node2 2017-08-28 15:32:22,109 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}' to Node2 2017-08-28 15:32:22,110 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 7) from Node4:0 2017-08-28 15:32:22,110 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,110 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,110 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:22,110 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:22,111 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:32:22,111 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node3:1 2017-08-28 15:32:22,111 | INFO | replica.py (1867) | enqueue_prepare | Node6:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} from Node3:1 2017-08-28 15:32:22,111 | DEBUG | replica.py (1969) | _request_pre_prepare_if_possible | Node6:1 not requesting a PRE-PREPARE because does not have sufficient PREPAREs for (2, 7) 2017-08-28 15:32:22,111 | DEBUG | replica.py ( 881) | processPrepare | Node6:1 cannot process incoming PREPARE 2017-08-28 15:32:22,111 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node2:1 2017-08-28 15:32:22,143 | INFO | replica.py (1867) | enqueue_prepare | Node6:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} from Node2:1 2017-08-28 15:32:22,152 | DEBUG | replica.py (1969) | _request_pre_prepare_if_possible | Node6:1 not requesting a PRE-PREPARE because does not have sufficient PREPAREs for (2, 7) 2017-08-28 15:32:22,154 | DEBUG | replica.py ( 881) | processPrepare | Node6:1 cannot process incoming PREPARE 2017-08-28 15:32:22,162 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:32:22,171 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,172 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503934341, 'instId': 1, 'ppSeqNo': 7, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:32:22,172 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,182 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,183 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppSeqNo': 7, 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:32:22,183 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,183 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:32:22,191 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:32:22,192 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934341,"ppSeqNo":7,"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:32:22,200 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,201 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'op': 'PREPARE', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ppSeqNo': 7}, 'Node5') 2017-08-28 15:32:22,206 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,207 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,207 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'ppSeqNo': 7}, 'Node5') 2017-08-28 15:32:22,207 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,239 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 7) from Node2:0 2017-08-28 15:32:22,246 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,246 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,246 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 7) from Node5:0 2017-08-28 15:32:22,247 | DEBUG | replica.py ( 953) | doCommit | Node6:0 Sending COMMIT(2, 7) at 250638.395837021 2017-08-28 15:32:22,254 | DEBUG | replica.py (2133) | send | Node6:0 sending Commit 2017-08-28 15:32:22,254 | TRACE | replica.py (2134) | send | Node6:0 sending COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,255 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,255 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,264 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:22,264 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:22,264 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 0) 2017-08-28 15:32:22,264 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node1:1 2017-08-28 15:32:22,267 | INFO | replica.py (1867) | enqueue_prepare | Node6:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} from Node1:1 2017-08-28 15:32:22,277 | DEBUG | replica.py (1969) | _request_pre_prepare_if_possible | Node6:1 not requesting a PRE-PREPARE because does not have sufficient PREPAREs for (2, 7) 2017-08-28 15:32:22,277 | DEBUG | replica.py ( 881) | processPrepare | Node6:1 cannot process incoming PREPARE 2017-08-28 15:32:22,277 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node5:1 2017-08-28 15:32:22,278 | INFO | replica.py (1867) | enqueue_prepare | Node6:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} from Node5:1 2017-08-28 15:32:22,278 | DEBUG | replica.py (1979) | _request_pre_prepare_if_possible | Node6:1 not requesting a PRE-PREPARE since already found stashed for (2, 7) 2017-08-28 15:32:22,288 | DEBUG | replica.py ( 881) | processPrepare | Node6:1 cannot process incoming PREPARE 2017-08-28 15:32:22,289 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:22,304 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,309 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppTime': 1503934341, 'instId': 0, 'ppSeqNo': 7, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:32:22,309 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,330 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:22,331 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 7) from Node1:0 2017-08-28 15:32:22,341 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 7, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:22,341 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,342 | WARNING | replica.py ( 983) | __is_next_pre_prepare | Node6:1 missing PRE-PREPAREs between 7 and 0 2017-08-28 15:32:22,347 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node6:1 Setting last ordered for non-master as (2, 0) 2017-08-28 15:32:22,348 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 6) 2017-08-28 15:32:22,358 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:22,358 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":7,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":7,"instId":1,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:22,370 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":7,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":7,"instId":1,"viewNo":2,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:32:22,370 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":7,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":7,"instId":1,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:22,371 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,371 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:32:22,379 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,385 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,385 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:32:22,385 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,404 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node5 2017-08-28 15:32:22,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node5 2017-08-28 15:32:22,412 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node4 2017-08-28 15:32:22,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node4 2017-08-28 15:32:22,421 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node1 2017-08-28 15:32:22,422 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node1 2017-08-28 15:32:22,431 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node3 2017-08-28 15:32:22,432 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node3 2017-08-28 15:32:22,438 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node2 2017-08-28 15:32:22,438 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}' to Node2 2017-08-28 15:32:22,447 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 7) from Node4:0 2017-08-28 15:32:22,456 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,457 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,457 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:1 received PRE-PREPARE(2, 7) from Node4:1 at 250638.606152761 2017-08-28 15:32:22,457 | DEBUG | replica.py (1885) | dequeue_prepares | Node6:1 popping stashed PREPARE(2, 7) 2017-08-28 15:32:22,468 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node3:1 2017-08-28 15:32:22,468 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,468 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,474 | DEBUG | replica.py (1885) | dequeue_prepares | Node6:1 popping stashed PREPARE(2, 7) 2017-08-28 15:32:22,474 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node2:1 2017-08-28 15:32:22,474 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,475 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,475 | DEBUG | replica.py (1885) | dequeue_prepares | Node6:1 popping stashed PREPARE(2, 7) 2017-08-28 15:32:22,484 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node1:1 2017-08-28 15:32:22,491 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,492 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,492 | DEBUG | replica.py (1885) | dequeue_prepares | Node6:1 popping stashed PREPARE(2, 7) 2017-08-28 15:32:22,492 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 7) from Node5:1 2017-08-28 15:32:22,493 | DEBUG | replica.py ( 953) | doCommit | Node6:1 Sending COMMIT(2, 7) at 250638.641722142 2017-08-28 15:32:22,502 | DEBUG | replica.py (2133) | send | Node6:1 sending Commit 2017-08-28 15:32:22,502 | TRACE | replica.py (2134) | send | Node6:1 sending COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,503 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,503 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 7) 2017-08-28 15:32:22,511 | DEBUG | replica.py (1891) | dequeue_prepares | Node6:1 processed 4 PREPAREs waiting for PRE-PREPARE for view no 2 and seq no 7 2017-08-28 15:32:22,512 | DEBUG | replica.py ( 934) | doPrepare | Node6:1 Sending PREPARE(2, 7) at 250638.660855598 2017-08-28 15:32:22,512 | DEBUG | replica.py (2133) | send | Node6:1 sending Prepare 2017-08-28 15:32:22,512 | TRACE | replica.py (2134) | send | Node6:1 sending PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,518 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} 2017-08-28 15:32:22,519 | DEBUG | replica.py ( 612) | trackBatches | Node6:1 tracking batch for PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'reqIdr': [('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353)], 'ppTime': 1503934341, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} with state root None 2017-08-28 15:32:22,519 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:1 processed incoming PRE-PREPARE(2, 7) 2017-08-28 15:32:22,526 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 7) from Node4:1 2017-08-28 15:32:22,527 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,527 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,538 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:32:22,547 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":1}']} 2017-08-28 15:32:22,548 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:32:22,548 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":7,"instId":1}']} 2017-08-28 15:32:22,548 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,557 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 0, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:32:22,558 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,558 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,571 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 1, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:32:22,572 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,572 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}']} 2017-08-28 15:32:22,578 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:32:22,578 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}']} 2017-08-28 15:32:22,579 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,583 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:32:22,583 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,586 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,586 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:32:22,587 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,595 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:22,596 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:32:22,596 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":7,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:22,604 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,604 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:32:22,611 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,611 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,616 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:32:22,616 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,616 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}']} 2017-08-28 15:32:22,623 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:32:22,624 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":7}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":7}']} 2017-08-28 15:32:22,624 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,624 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:32:22,636 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,638 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,638 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 7, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:32:22,639 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} 2017-08-28 15:32:22,647 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:22,673 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': 'f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17', 'ppTime': 1503934341, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 7, 'stateRootHash': None} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:22,673 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 7) from Node3:0 2017-08-28 15:32:22,673 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,684 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,685 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 7) from Node1:0 2017-08-28 15:32:22,685 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,685 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,685 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 7) from Node5:0 2017-08-28 15:32:22,707 | TRACE | replica.py ( 925) | tryOrder | Node6:0 returning request to node 2017-08-28 15:32:22,707 | DEBUG | replica.py (1437) | doOrder | Node6:0 ordering COMMIT (2, 7) 2017-08-28 15:32:22,707 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:0 set last ordered as (2, 7) 2017-08-28 15:32:22,708 | DEBUG | replica.py (2133) | send | Node6:0 sending Ordered 2017-08-28 15:32:22,708 | TRACE | replica.py (2134) | send | Node6:0 sending ORDERED{'viewNo': 2, 'ppSeqNo': 7, 'ppTime': 1503934341, 'instId': 0, 'reqIdr': [], 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:22,708 | DEBUG | replica.py (1469) | order_3pc_key | Node6:0 ordered request (2, 7) 2017-08-28 15:32:22,709 | DEBUG | replica.py (1540) | _newCheckpointState | Node6:0 adding new checkpoint state for (7, 106) 2017-08-28 15:32:22,709 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,709 | TRACE | message_processor.py ( 29) | discard | Node6:0 discarding message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 7} because already ordered 3 phase message 2017-08-28 15:32:22,718 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 7) from Node3:1 2017-08-28 15:32:22,723 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,724 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,724 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 7) from Node1:1 2017-08-28 15:32:22,724 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} commits where f is 1 2017-08-28 15:32:22,725 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,730 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 7) from Node5:1 2017-08-28 15:32:22,730 | TRACE | replica.py ( 925) | tryOrder | Node6:1 returning request to node 2017-08-28 15:32:22,730 | DEBUG | replica.py (1437) | doOrder | Node6:1 ordering COMMIT (2, 7) 2017-08-28 15:32:22,735 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 7) 2017-08-28 15:32:22,737 | DEBUG | replica.py (2133) | send | Node6:1 sending Ordered 2017-08-28 15:32:22,739 | TRACE | replica.py (2134) | send | Node6:1 sending ORDERED{'viewNo': 2, 'ppSeqNo': 7, 'ppTime': 1503934341, 'instId': 1, 'reqIdr': [('XhYtvJqezMUKfF6KVNaGmT', 1503934341241353)], 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:22,744 | DEBUG | replica.py (1469) | order_3pc_key | Node6:1 ordered request (2, 7) 2017-08-28 15:32:22,745 | DEBUG | replica.py (1540) | _newCheckpointState | Node6:1 adding new checkpoint state for (7, 106) 2017-08-28 15:32:22,745 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 7) 2017-08-28 15:32:22,745 | TRACE | message_processor.py ( 29) | discard | Node6:1 discarding message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 7} because already ordered 3 phase message 2017-08-28 15:32:22,762 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node5 into one transmission 2017-08-28 15:32:22,776 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":1,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}', b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}']) 2017-08-28 15:32:22,776 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' 2017-08-28 15:32:22,777 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' to Node5 2017-08-28 15:32:22,777 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node4 into one transmission 2017-08-28 15:32:22,791 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":1,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}', b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}']) 2017-08-28 15:32:22,792 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' 2017-08-28 15:32:22,795 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' to Node4 2017-08-28 15:32:22,795 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node1 into one transmission 2017-08-28 15:32:22,799 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":1,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}', b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}']) 2017-08-28 15:32:22,799 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' 2017-08-28 15:32:22,800 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' to Node1 2017-08-28 15:32:22,800 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node3 into one transmission 2017-08-28 15:32:22,811 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":1,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}', b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}']) 2017-08-28 15:32:22,812 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node3: b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' 2017-08-28 15:32:22,819 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' to Node3 2017-08-28 15:32:22,823 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node2 into one transmission 2017-08-28 15:32:22,824 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":1,"viewNo":2,"ppSeqNo":7,"op":"COMMIT"}', b'{"digest":"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934341,"viewNo":2,"ppSeqNo":7}']) 2017-08-28 15:32:22,824 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node2: b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' 2017-08-28 15:32:22,824 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":7,\\"op\\":\\"COMMIT\\"}","{\\"digest\\":\\"f4083a5551140ae7e2ce519b4ba3a2e73265a576ea7afca97a1510e5596f4b17\\",\\"txnRootHash\\":null,\\"stateRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"instId\\":1,\\"ppTime\\":1503934341,\\"viewNo\\":2,\\"ppSeqNo\\":7}"],"op":"BATCH"}' to Node2 2017-08-28 15:32:22,840 | DEBUG | node.py (1828) | processOrdered | Node6 executing Ordered batch 2 7 of 0 requests 2017-08-28 15:32:22,840 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:32:22,851 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:22,852 | TRACE | node.py (1839) | processOrdered | Node6 got ordered requests from backup replica 1 2017-08-28 15:32:22,887 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.084 seconds 2017-08-28 15:32:26,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:32:26,932 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:32:26,933 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:32:26,934 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:32:26,935 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:32:26,939 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:32:29,464 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 110 2017-08-28 15:32:29,464 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:29,466 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:29,467 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:29,467 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:29,471 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node6 found master had no record yet for XhYtvJqezMUKfF6KVNaGmT 2017-08-28 15:32:29,473 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:29,478 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 111 to run in 10 seconds 2017-08-28 15:32:39,479 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 111 2017-08-28 15:32:39,502 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:39,502 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:39,503 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:39,503 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:39,507 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node6 found master had no record yet for XhYtvJqezMUKfF6KVNaGmT 2017-08-28 15:32:39,508 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:39,510 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 112 to run in 10 seconds 2017-08-28 15:32:41,951 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:32:41,951 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:32:41,955 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:32:41,956 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:32:41,956 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:32:41,959 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:32:49,539 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 112 2017-08-28 15:32:49,565 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:49,566 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:49,567 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:49,594 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:49,595 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node6 found master had no record yet for XhYtvJqezMUKfF6KVNaGmT 2017-08-28 15:32:49,598 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:49,598 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 113 to run in 10 seconds 2017-08-28 15:32:49,637 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:32:49,647 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on request 1503934369465182 2017-08-28 15:32:49,648 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,656 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,657 | DEBUG | node.py (1690) | processRequest | Node6 received client request: SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:32:49,668 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,677 | DEBUG | propagator.py ( 148) | propagate | Node6 propagating FBQ5W7LxRp8bLeVNEafjpd request 1503934369465182 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:32:49,677 | DEBUG | node.py (2566) | send | Node6 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:49,685 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} to its replicas since not finalised 2017-08-28 15:32:49,686 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934369465182,"op":"REQACK","identifier":"FBQ5W7LxRp8bLeVNEafjpd"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:32:49,697 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:32:49,708 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:32:49,712 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:32:49,723 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:32:49,730 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:32:49,731 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:32:49,740 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:32:49,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:32:49,756 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:32:49,757 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss","reqId":1503934369465182,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:32:49,776 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:32:49,793 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on propagate request 1503934369465182 2017-08-28 15:32:49,799 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,800 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, 'op': 'PROPAGATE'}, 'Node4') 2017-08-28 15:32:49,809 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,810 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on propagate request 1503934369465182 2017-08-28 15:32:49,821 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, 'op': 'PROPAGATE'}, 'Node1') 2017-08-28 15:32:49,831 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,837 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on propagate request 1503934369465182 2017-08-28 15:32:49,838 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,846 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, 'op': 'PROPAGATE'}, 'Node2') 2017-08-28 15:32:49,847 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,862 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on propagate request 1503934369465182 2017-08-28 15:32:49,863 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,870 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, 'op': 'PROPAGATE'}, 'Node5') 2017-08-28 15:32:49,871 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,881 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated FBQ5W7LxRp8bLeVNEafjpd signature on propagate request 1503934369465182 2017-08-28 15:32:49,891 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,893 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, 'op': 'PROPAGATE'}, 'Node3') 2017-08-28 15:32:49,894 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,908 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,917 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,926 | DEBUG | propagator.py ( 213) | forward | Node6 forwarding request ('FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182) to 0 replicas 2017-08-28 15:32:49,927 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,936 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,937 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} to its replicas since already forwarded 2017-08-28 15:32:49,951 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,953 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,965 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} to its replicas since already forwarded 2017-08-28 15:32:49,966 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:49,966 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:49,983 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} to its replicas since already forwarded 2017-08-28 15:32:49,988 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}} 2017-08-28 15:32:50,001 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} 2017-08-28 15:32:50,002 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'} to its replicas since already forwarded 2017-08-28 15:32:50,062 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.526 seconds 2017-08-28 15:32:50,102 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:50,114 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'ppTime': 1503934370, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:50,123 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'discarded': 1, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'ppSeqNo': 8, 'instId': 1, 'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'stateRootHash': None, 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node4') 2017-08-28 15:32:50,123 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'ppTime': 1503934370, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:50,146 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:1 received PRE-PREPARE(2, 8) from Node4:1 at 250666.295320085 2017-08-28 15:32:50,155 | DEBUG | replica.py ( 934) | doPrepare | Node6:1 Sending PREPARE(2, 8) at 250666.304449765 2017-08-28 15:32:50,156 | DEBUG | replica.py (2133) | send | Node6:1 sending Prepare 2017-08-28 15:32:50,156 | TRACE | replica.py (2134) | send | Node6:1 sending PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,156 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,156 | DEBUG | replica.py ( 612) | trackBatches | Node6:1 tracking batch for PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [('FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182)], 'ppTime': 1503934370, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} with state root None 2017-08-28 15:32:50,156 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:1 processed incoming PRE-PREPARE(2, 8) 2017-08-28 15:32:50,157 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:50,158 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'ppTime': 1503934370, 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:50,158 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'instId': 0, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'discarded': 0, 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'viewNo': 2, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node3') 2017-08-28 15:32:50,158 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [['FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182]], 'ppTime': 1503934370, 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:50,197 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:50,219 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:0 received PRE-PREPARE(2, 8) from Node3:0 at 250666.368196302 2017-08-28 15:32:50,220 | DEBUG | replica.py (1034) | validate_pre_prepare | Node6:0 state root before processing PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [('FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182)], 'ppTime': 1503934370, 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} is b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9', 67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ 2017-08-28 15:32:50,221 | WARNING | replica.py ( 655) | processReqDuringBatch | Node6:0 encountered exception UnauthorizedClientRequest('FBQ5W7LxRp8bLeVNEafjpd is not a steward of node 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc',) while processing Request: {'signature': '5rdFRSA94d7UYhzsY4rrUSM3taAEFoxpgz39vYz6eZdd9kXZJMH5hpiHTUxnWQr1vvrKAARqKUei5FHhYaLzbsss', 'reqId': 1503934369465182, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'FBQ5W7LxRp8bLeVNEafjpd'}, will reject 2017-08-28 15:32:50,232 | DEBUG | replica.py ( 934) | doPrepare | Node6:0 Sending PREPARE(2, 8) at 250666.380834438 2017-08-28 15:32:50,232 | DEBUG | replica.py (2133) | send | Node6:0 sending Prepare 2017-08-28 15:32:50,242 | TRACE | replica.py (2134) | send | Node6:0 sending PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,243 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,249 | DEBUG | replica.py ( 612) | trackBatches | Node6:0 tracking batch for PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'reqIdr': [('FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182)], 'ppTime': 1503934370, 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} with state root b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9' 2017-08-28 15:32:50,249 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:0 processed incoming PRE-PREPARE(2, 8) 2017-08-28 15:32:50,283 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node5 2017-08-28 15:32:50,289 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node5 2017-08-28 15:32:50,289 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node4 2017-08-28 15:32:50,290 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node4 2017-08-28 15:32:50,296 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node1 2017-08-28 15:32:50,297 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node1 2017-08-28 15:32:50,297 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node3 2017-08-28 15:32:50,298 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node3 2017-08-28 15:32:50,316 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node2 2017-08-28 15:32:50,316 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node2 2017-08-28 15:32:50,335 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934369465182,"reason":"client request invalid: UnauthorizedClientRequest(\'FBQ5W7LxRp8bLeVNEafjpd is not a steward of node 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc\',)","op":"REJECT","identifier":"FBQ5W7LxRp8bLeVNEafjpd"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:32:50,350 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:50,362 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:32:50,374 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,374 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 8, 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node4') 2017-08-28 15:32:50,374 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,384 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,385 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'viewNo': 2, 'ppSeqNo': 8, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node3') 2017-08-28 15:32:50,385 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,406 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node5 2017-08-28 15:32:50,406 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node5 2017-08-28 15:32:50,407 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node4 2017-08-28 15:32:50,407 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node4 2017-08-28 15:32:50,416 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node1 2017-08-28 15:32:50,416 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node1 2017-08-28 15:32:50,423 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node3 2017-08-28 15:32:50,426 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node3 2017-08-28 15:32:50,427 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node2 2017-08-28 15:32:50,437 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934370,"viewNo":2,"ppSeqNo":8}' to Node2 2017-08-28 15:32:50,438 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 8) from Node4:0 2017-08-28 15:32:50,453 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,453 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,453 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 8) from Node3:1 2017-08-28 15:32:50,454 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,462 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,475 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:32:50,477 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,477 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503934370, 'instId': 1, 'ppSeqNo': 8, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:32:50,477 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,477 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}']} 2017-08-28 15:32:50,477 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:32:50,477 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":8,"ppTime":1503934370,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89"}']} 2017-08-28 15:32:50,478 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,478 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppSeqNo': 8, 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:32:50,478 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,478 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,497 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppSeqNo': 8, 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:32:50,497 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,515 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:32:50,515 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:32:50,515 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934370,"ppSeqNo":8,"digest":"a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:32:50,525 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,525 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'op': 'PREPARE', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ppSeqNo': 8}, 'Node5') 2017-08-28 15:32:50,525 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,537 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,537 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'ppSeqNo': 8}, 'Node5') 2017-08-28 15:32:50,537 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,557 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 8) from Node2:0 2017-08-28 15:32:50,572 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,572 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,572 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 8) from Node5:0 2017-08-28 15:32:50,580 | DEBUG | replica.py ( 953) | doCommit | Node6:0 Sending COMMIT(2, 8) at 250666.728753633 2017-08-28 15:32:50,580 | DEBUG | replica.py (2133) | send | Node6:0 sending Commit 2017-08-28 15:32:50,580 | TRACE | replica.py (2134) | send | Node6:0 sending COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,588 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,588 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,588 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 8) from Node1:1 2017-08-28 15:32:50,602 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,603 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,603 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 8) from Node2:1 2017-08-28 15:32:50,607 | DEBUG | replica.py ( 953) | doCommit | Node6:1 Sending COMMIT(2, 8) at 250666.756614292 2017-08-28 15:32:50,608 | DEBUG | replica.py (2133) | send | Node6:1 sending Commit 2017-08-28 15:32:50,608 | TRACE | replica.py (2134) | send | Node6:1 sending COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,608 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,609 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,616 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 8) from Node5:1 2017-08-28 15:32:50,616 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 8, 'stateRootHash': None} 2017-08-28 15:32:50,627 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,634 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:32:50,635 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,645 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppTime': 1503934370, 'instId': 0, 'ppSeqNo': 8, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:32:50,646 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,661 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:50,671 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:32:50,671 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 8) from Node1:0 2017-08-28 15:32:50,680 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': 'a59b7a209feecc73354aa1179af0065f6a0b7a2e894f20ee74302ef8031d1e89', 'ppTime': 1503934370, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 8, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:32:50,680 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 8) 2017-08-28 15:32:50,681 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:32:50,682 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,682 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:32:50,682 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,683 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,683 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 1, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:32:50,683 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,710 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node5 into one transmission 2017-08-28 15:32:50,722 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}']) 2017-08-28 15:32:50,722 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:32:50,729 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node5 2017-08-28 15:32:50,729 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node4 into one transmission 2017-08-28 15:32:50,729 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}']) 2017-08-28 15:32:50,730 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:32:50,739 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node4 2017-08-28 15:32:50,745 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node1 into one transmission 2017-08-28 15:32:50,745 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}']) 2017-08-28 15:32:50,745 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:32:50,746 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node1 2017-08-28 15:32:50,757 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node3 into one transmission 2017-08-28 15:32:50,757 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}']) 2017-08-28 15:32:50,757 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node3: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:32:50,758 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node3 2017-08-28 15:32:50,769 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node2 into one transmission 2017-08-28 15:32:50,769 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":8,"op":"COMMIT"}']) 2017-08-28 15:32:50,769 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node2: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:32:50,775 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":8,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node2 2017-08-28 15:32:50,780 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 8) from Node4:1 2017-08-28 15:32:50,789 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,789 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,790 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 8) from Node3:1 2017-08-28 15:32:50,797 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,797 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,798 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:32:50,810 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:50,819 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:32:50,819 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":8,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:32:50,820 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,825 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:32:50,826 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,826 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,826 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:32:50,835 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,838 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}']} 2017-08-28 15:32:50,838 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:32:50,838 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}']} 2017-08-28 15:32:50,839 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,847 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:32:50,850 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,850 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,851 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:32:50,861 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,861 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,861 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 0, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:32:50,871 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,871 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}']} 2017-08-28 15:32:50,876 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:32:50,877 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":8}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":8}']} 2017-08-28 15:32:50,877 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,877 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:32:50,891 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,892 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,892 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:32:50,895 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,896 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,896 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 8, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:32:50,896 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} 2017-08-28 15:32:50,930 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 8) from Node5:0 2017-08-28 15:32:50,930 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,930 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,931 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 8) from Node2:0 2017-08-28 15:32:50,939 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,940 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,940 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 8) from Node3:0 2017-08-28 15:32:50,948 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,949 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,949 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 8) from Node1:0 2017-08-28 15:32:50,949 | TRACE | replica.py ( 925) | tryOrder | Node6:0 returning request to node 2017-08-28 15:32:50,958 | DEBUG | replica.py (1437) | doOrder | Node6:0 ordering COMMIT (2, 8) 2017-08-28 15:32:50,958 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:0 set last ordered as (2, 8) 2017-08-28 15:32:50,959 | DEBUG | replica.py (2133) | send | Node6:0 sending Ordered 2017-08-28 15:32:50,971 | TRACE | replica.py (2134) | send | Node6:0 sending ORDERED{'viewNo': 2, 'ppSeqNo': 8, 'ppTime': 1503934370, 'instId': 0, 'reqIdr': [], 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:32:50,971 | DEBUG | replica.py (1469) | order_3pc_key | Node6:0 ordered request (2, 8) 2017-08-28 15:32:50,972 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,972 | TRACE | message_processor.py ( 29) | discard | Node6:0 discarding message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 8} because already ordered 3 phase message 2017-08-28 15:32:50,973 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 8) from Node5:1 2017-08-28 15:32:50,979 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} commits where f is 1 2017-08-28 15:32:50,980 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 8) 2017-08-28 15:32:50,980 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 8) from Node2:1 2017-08-28 15:32:50,991 | TRACE | replica.py ( 925) | tryOrder | Node6:1 returning request to node 2017-08-28 15:32:50,991 | DEBUG | replica.py (1437) | doOrder | Node6:1 ordering COMMIT (2, 8) 2017-08-28 15:32:50,994 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 8) 2017-08-28 15:32:50,994 | DEBUG | replica.py (2133) | send | Node6:1 sending Ordered 2017-08-28 15:32:50,995 | TRACE | replica.py (2134) | send | Node6:1 sending ORDERED{'viewNo': 2, 'ppSeqNo': 8, 'ppTime': 1503934370, 'instId': 1, 'reqIdr': [('FBQ5W7LxRp8bLeVNEafjpd', 1503934369465182)], 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:32:50,995 | DEBUG | replica.py (1469) | order_3pc_key | Node6:1 ordered request (2, 8) 2017-08-28 15:32:51,003 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 8) 2017-08-28 15:32:51,004 | TRACE | message_processor.py ( 29) | discard | Node6:1 discarding message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 8} because already ordered 3 phase message 2017-08-28 15:32:51,032 | DEBUG | node.py (1828) | processOrdered | Node6 executing Ordered batch 2 8 of 0 requests 2017-08-28 15:32:51,032 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:32:51,039 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:51,044 | TRACE | node.py (1839) | processOrdered | Node6 got ordered requests from backup replica 1 2017-08-28 15:32:51,092 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.992 seconds 2017-08-28 15:32:56,978 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:32:56,984 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:32:56,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:32:56,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:32:56,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:32:56,993 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:32:59,613 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 113 2017-08-28 15:32:59,615 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:32:59,616 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:32:59,618 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node6 master throughput is not measurable. 2017-08-28 15:32:59,618 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:32:59,627 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node6 found master had no record yet for FBQ5W7LxRp8bLeVNEafjpd 2017-08-28 15:32:59,627 | DEBUG | node.py (1976) | checkPerformance | Node6's master has higher performance than backups 2017-08-28 15:32:59,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 114 to run in 10 seconds 2017-08-28 15:33:05,919 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:33:05,926 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on request 1503934385861338 2017-08-28 15:33:05,936 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:05,937 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:05,937 | DEBUG | node.py (1690) | processRequest | Node6 received client request: SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:33:05,937 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:05,938 | DEBUG | propagator.py ( 148) | propagate | Node6 propagating 6XiqM9DwYDgFnmyQQBFr72 request 1503934385861338 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:33:05,938 | DEBUG | node.py (2566) | send | Node6 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:05,938 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} to its replicas since not finalised 2017-08-28 15:33:05,938 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934385861338,"op":"REQACK","identifier":"6XiqM9DwYDgFnmyQQBFr72"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:33:05,972 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:33:05,980 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:33:05,991 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:33:05,992 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:33:06,003 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:33:06,013 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:33:06,014 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:33:06,023 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:33:06,032 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:33:06,032 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw","reqId":1503934385861338,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"6XiqM9DwYDgFnmyQQBFr72"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:33:06,056 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:33:06,060 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503934385861338 2017-08-28 15:33:06,073 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,073 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, 'op': 'PROPAGATE'}, 'Node4') 2017-08-28 15:33:06,075 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,086 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503934385861338 2017-08-28 15:33:06,097 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,097 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, 'op': 'PROPAGATE'}, 'Node2') 2017-08-28 15:33:06,103 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,105 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503934385861338 2017-08-28 15:33:06,116 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,117 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, 'op': 'PROPAGATE'}, 'Node1') 2017-08-28 15:33:06,127 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,132 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503934385861338 2017-08-28 15:33:06,141 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,142 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, 'op': 'PROPAGATE'}, 'Node3') 2017-08-28 15:33:06,150 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,152 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503934385861338 2017-08-28 15:33:06,164 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,171 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, 'op': 'PROPAGATE'}, 'Node5') 2017-08-28 15:33:06,176 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,177 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,194 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:06,194 | DEBUG | propagator.py ( 213) | forward | Node6 forwarding request ('6XiqM9DwYDgFnmyQQBFr72', 1503934385861338) to 0 replicas 2017-08-28 15:33:06,195 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,210 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:06,211 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} to its replicas since already forwarded 2017-08-28 15:33:06,221 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,222 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:06,233 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} to its replicas since already forwarded 2017-08-28 15:33:06,240 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,240 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:06,252 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} to its replicas since already forwarded 2017-08-28 15:33:06,252 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:33:06,268 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} 2017-08-28 15:33:06,269 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'} to its replicas since already forwarded 2017-08-28 15:33:06,338 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.432 seconds 2017-08-28 15:33:06,384 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:33:06,396 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'ppTime': 1503934386, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:06,397 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'discarded': 1, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'ppSeqNo': 9, 'instId': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'stateRootHash': None, 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node4') 2017-08-28 15:33:06,397 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'ppTime': 1503934386, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:06,417 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:1 received PRE-PREPARE(2, 9) from Node4:1 at 250682.565783403 2017-08-28 15:33:06,428 | DEBUG | replica.py ( 934) | doPrepare | Node6:1 Sending PREPARE(2, 9) at 250682.577578278 2017-08-28 15:33:06,429 | DEBUG | replica.py (2133) | send | Node6:1 sending Prepare 2017-08-28 15:33:06,439 | TRACE | replica.py (2134) | send | Node6:1 sending PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,439 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,449 | DEBUG | replica.py ( 612) | trackBatches | Node6:1 tracking batch for PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503934385861338)], 'ppTime': 1503934386, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} with state root None 2017-08-28 15:33:06,449 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:1 processed incoming PRE-PREPARE(2, 9) 2017-08-28 15:33:06,450 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:33:06,458 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'ppTime': 1503934386, 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:06,467 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'instId': 0, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'discarded': 0, 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'viewNo': 2, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node3') 2017-08-28 15:33:06,468 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503934385861338]], 'ppTime': 1503934386, 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:06,485 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:06,501 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:0 received PRE-PREPARE(2, 9) from Node3:0 at 250682.649844996 2017-08-28 15:33:06,502 | DEBUG | replica.py (1034) | validate_pre_prepare | Node6:0 state root before processing PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503934385861338)], 'ppTime': 1503934386, 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} is b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9', 67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ 2017-08-28 15:33:06,508 | WARNING | replica.py ( 655) | processReqDuringBatch | Node6:0 encountered exception UnauthorizedClientRequest('6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot update a node',) while processing Request: {'signature': '674RWKwdAoxi8aY6vuUifCByKbsXYcZ9ogBRAT4DSnve6k4RcRf6zngXCqZYEk1ULZRs5MBvehmGV5dvYxu17Qzw', 'reqId': 1503934385861338, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}, will reject 2017-08-28 15:33:06,519 | DEBUG | replica.py ( 934) | doPrepare | Node6:0 Sending PREPARE(2, 9) at 250682.667706289 2017-08-28 15:33:06,519 | DEBUG | replica.py (2133) | send | Node6:0 sending Prepare 2017-08-28 15:33:06,528 | TRACE | replica.py (2134) | send | Node6:0 sending PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,528 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,532 | DEBUG | replica.py ( 612) | trackBatches | Node6:0 tracking batch for PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503934385861338)], 'ppTime': 1503934386, 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} with state root b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9' 2017-08-28 15:33:06,532 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:0 processed incoming PRE-PREPARE(2, 9) 2017-08-28 15:33:06,558 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node5 2017-08-28 15:33:06,559 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node5 2017-08-28 15:33:06,573 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node4 2017-08-28 15:33:06,573 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node4 2017-08-28 15:33:06,574 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node1 2017-08-28 15:33:06,576 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node1 2017-08-28 15:33:06,576 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node3 2017-08-28 15:33:06,576 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node3 2017-08-28 15:33:06,586 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node2 2017-08-28 15:33:06,587 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node2 2017-08-28 15:33:06,594 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934385861338,"reason":"client request invalid: UnauthorizedClientRequest(\'6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot update a node\',)","op":"REJECT","identifier":"6XiqM9DwYDgFnmyQQBFr72"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:33:06,605 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:06,606 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:33:06,621 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,621 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'viewNo': 2, 'ppSeqNo': 9, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node3') 2017-08-28 15:33:06,622 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,631 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,636 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 9, 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node4') 2017-08-28 15:33:06,637 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,653 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node5 2017-08-28 15:33:06,663 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node5 2017-08-28 15:33:06,664 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node4 2017-08-28 15:33:06,664 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node4 2017-08-28 15:33:06,679 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node1 2017-08-28 15:33:06,679 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node1 2017-08-28 15:33:06,691 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node3 2017-08-28 15:33:06,692 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node3 2017-08-28 15:33:06,692 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node2 2017-08-28 15:33:06,692 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934386,"viewNo":2,"ppSeqNo":9}' to Node2 2017-08-28 15:33:06,710 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 9) from Node4:0 2017-08-28 15:33:06,727 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,727 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,728 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 9) from Node3:1 2017-08-28 15:33:06,728 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,737 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,738 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:33:06,750 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}']} 2017-08-28 15:33:06,762 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:33:06,762 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","instId":0,"viewNo":2,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}', '{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":9,"ppTime":1503934386,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d"}']} 2017-08-28 15:33:06,771 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,772 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppSeqNo': 9, 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:33:06,772 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,772 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,781 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppSeqNo': 9, 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:33:06,781 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,782 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:33:06,791 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:33:06,792 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","viewNo":2,"op":"PREPARE","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ"}', '{"instId":1,"ppTime":1503934386,"ppSeqNo":9,"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","stateRootHash":null,"viewNo":2,"op":"PREPARE","txnRootHash":null}']} 2017-08-28 15:33:06,802 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,802 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'op': 'PREPARE', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ppSeqNo': 9}, 'Node5') 2017-08-28 15:33:06,802 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,803 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,813 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'ppSeqNo': 9}, 'Node5') 2017-08-28 15:33:06,814 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,814 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","ppTime":1503934386,"ppSeqNo":9,"instId":0,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2}', '{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":null,"ppTime":1503934386,"ppSeqNo":9,"instId":1,"txnRootHash":null,"viewNo":2}']} 2017-08-28 15:33:06,825 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","ppTime":1503934386,"ppSeqNo":9,"instId":0,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2}', '{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":null,"ppTime":1503934386,"ppSeqNo":9,"instId":1,"txnRootHash":null,"viewNo":2}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:33:06,830 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","ppTime":1503934386,"ppSeqNo":9,"instId":0,"txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","viewNo":2}', '{"digest":"01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d","op":"PREPARE","stateRootHash":null,"ppTime":1503934386,"ppSeqNo":9,"instId":1,"txnRootHash":null,"viewNo":2}']} 2017-08-28 15:33:06,830 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,830 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppTime': 1503934386, 'instId': 0, 'ppSeqNo': 9, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:33:06,830 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,831 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,831 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503934386, 'instId': 1, 'ppSeqNo': 9, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:33:06,831 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,873 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 9) from Node2:0 2017-08-28 15:33:06,882 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,882 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,889 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 9) from Node5:0 2017-08-28 15:33:06,889 | DEBUG | replica.py ( 953) | doCommit | Node6:0 Sending COMMIT(2, 9) at 250683.038410596 2017-08-28 15:33:06,890 | DEBUG | replica.py (2133) | send | Node6:0 sending Commit 2017-08-28 15:33:06,890 | TRACE | replica.py (2134) | send | Node6:0 sending COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:06,899 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:06,900 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,900 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 9) from Node1:0 2017-08-28 15:33:06,901 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 9, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:06,910 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,910 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 9) from Node2:1 2017-08-28 15:33:06,911 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,911 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,922 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 9) from Node5:1 2017-08-28 15:33:06,922 | DEBUG | replica.py ( 953) | doCommit | Node6:1 Sending COMMIT(2, 9) at 250683.071493404 2017-08-28 15:33:06,923 | DEBUG | replica.py (2133) | send | Node6:1 sending Commit 2017-08-28 15:33:06,923 | TRACE | replica.py (2134) | send | Node6:1 sending COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:06,923 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:06,932 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,932 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 9) from Node1:1 2017-08-28 15:33:06,932 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': '01c43924d5705be935bda848f1a02d1ab151e9721aa0975e09db012d6637d84d', 'ppTime': 1503934386, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 9, 'stateRootHash': None} 2017-08-28 15:33:06,935 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 9) 2017-08-28 15:33:06,968 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:06,968 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:06,992 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node5 into one transmission 2017-08-28 15:33:06,992 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}']) 2017-08-28 15:33:07,001 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node5: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:33:07,001 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node5 2017-08-28 15:33:07,002 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node4 into one transmission 2017-08-28 15:33:07,013 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}']) 2017-08-28 15:33:07,013 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node4: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:33:07,014 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node4 2017-08-28 15:33:07,014 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node1 into one transmission 2017-08-28 15:33:07,026 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}']) 2017-08-28 15:33:07,026 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node1: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:33:07,033 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node1 2017-08-28 15:33:07,033 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node3 into one transmission 2017-08-28 15:33:07,033 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}']) 2017-08-28 15:33:07,043 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node3: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:33:07,044 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node3 2017-08-28 15:33:07,044 | DEBUG | batched.py ( 99) | flushOutBoxes | Node6 batching 2 msgs to Node2 into one transmission 2017-08-28 15:33:07,051 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}', b'{"instId":1,"viewNo":2,"ppSeqNo":9,"op":"COMMIT"}']) 2017-08-28 15:33:07,052 | TRACE | batched.py ( 108) | flushOutBoxes | Node6 sending payload to Node2: b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' 2017-08-28 15:33:07,052 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"signature":null,"messages":["{\\"instId\\":0,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":2,\\"ppSeqNo\\":9,\\"op\\":\\"COMMIT\\"}"],"op":"BATCH"}' to Node2 2017-08-28 15:33:07,062 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:33:07,074 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":9,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":9,"instId":1,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:33:07,082 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":9,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":9,"instId":1,"viewNo":2,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:33:07,082 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":9,"instId":0,"viewNo":2,"op":"COMMIT"}', '{"ppSeqNo":9,"instId":1,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:33:07,083 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,083 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:33:07,092 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,092 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,092 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:33:07,093 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,103 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":1}']} 2017-08-28 15:33:07,105 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:33:07,105 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":0}', '{"op":"COMMIT","viewNo":2,"ppSeqNo":9,"instId":1}']} 2017-08-28 15:33:07,106 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,106 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 0, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:33:07,114 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,114 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,121 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 1, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:33:07,121 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,121 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:33:07,122 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node5') 2017-08-28 15:33:07,133 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":9,"viewNo":2,"op":"COMMIT"}']} 2017-08-28 15:33:07,137 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,137 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:33:07,137 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,137 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,138 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:33:07,146 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,146 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}']} 2017-08-28 15:33:07,147 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}'], 'op': 'BATCH'}, 'Node1') 2017-08-28 15:33:07,154 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}']} 2017-08-28 15:33:07,154 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,155 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:33:07,155 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,162 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,163 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:33:07,163 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,164 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}']} 2017-08-28 15:33:07,174 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:33:07,175 | DEBUG | node.py (1254) | unpackNodeMsg | Node6 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":2,"ppSeqNo":9}', '{"instId":1,"op":"COMMIT","viewNo":2,"ppSeqNo":9}']} 2017-08-28 15:33:07,175 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,181 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:33:07,182 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,182 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,193 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 9, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:33:07,194 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} 2017-08-28 15:33:07,217 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 108 2017-08-28 15:33:07,217 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:33:07,222 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:07,222 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 115 to run in 60 seconds 2017-08-28 15:33:07,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 16 2017-08-28 15:33:07,223 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:33:07,223 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 17 to run in 60 seconds 2017-08-28 15:33:07,234 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 9) from Node4:0 2017-08-28 15:33:07,248 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,248 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,248 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 9) from Node3:0 2017-08-28 15:33:07,256 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,257 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,257 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 9) from Node5:0 2017-08-28 15:33:07,257 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,258 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,270 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 9) from Node1:0 2017-08-28 15:33:07,271 | TRACE | replica.py ( 925) | tryOrder | Node6:0 returning request to node 2017-08-28 15:33:07,271 | DEBUG | replica.py (1437) | doOrder | Node6:0 ordering COMMIT (2, 9) 2017-08-28 15:33:07,277 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:0 set last ordered as (2, 9) 2017-08-28 15:33:07,277 | DEBUG | replica.py (2133) | send | Node6:0 sending Ordered 2017-08-28 15:33:07,277 | TRACE | replica.py (2134) | send | Node6:0 sending ORDERED{'viewNo': 2, 'ppSeqNo': 9, 'ppTime': 1503934386, 'instId': 0, 'reqIdr': [], 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:07,285 | DEBUG | replica.py (1469) | order_3pc_key | Node6:0 ordered request (2, 9) 2017-08-28 15:33:07,286 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,286 | TRACE | message_processor.py ( 29) | discard | Node6:0 discarding message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 9} because already ordered 3 phase message 2017-08-28 15:33:07,294 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 9) from Node4:1 2017-08-28 15:33:07,295 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,295 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,304 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 9) from Node3:1 2017-08-28 15:33:07,304 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,305 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,308 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 9) from Node5:1 2017-08-28 15:33:07,316 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} commits where f is 1 2017-08-28 15:33:07,316 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,316 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 9) from Node1:1 2017-08-28 15:33:07,325 | TRACE | replica.py ( 925) | tryOrder | Node6:1 returning request to node 2017-08-28 15:33:07,325 | DEBUG | replica.py (1437) | doOrder | Node6:1 ordering COMMIT (2, 9) 2017-08-28 15:33:07,325 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 9) 2017-08-28 15:33:07,326 | DEBUG | replica.py (2133) | send | Node6:1 sending Ordered 2017-08-28 15:33:07,335 | TRACE | replica.py (2134) | send | Node6:1 sending ORDERED{'viewNo': 2, 'ppSeqNo': 9, 'ppTime': 1503934386, 'instId': 1, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503934385861338)], 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:07,335 | DEBUG | replica.py (1469) | order_3pc_key | Node6:1 ordered request (2, 9) 2017-08-28 15:33:07,336 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 9) 2017-08-28 15:33:07,336 | TRACE | message_processor.py ( 29) | discard | Node6:1 discarding message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 9} because already ordered 3 phase message 2017-08-28 15:33:07,357 | DEBUG | node.py (1828) | processOrdered | Node6 executing Ordered batch 2 9 of 0 requests 2017-08-28 15:33:07,371 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:33:07,372 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:33:07,372 | TRACE | node.py (1839) | processOrdered | Node6 got ordered requests from backup replica 1 2017-08-28 15:33:07,420 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.037 seconds 2017-08-28 15:33:09,661 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 114 2017-08-28 15:33:09,665 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:33:09,666 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:09,666 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:33:09,666 | DEBUG | throttler.py ( 31) | acquire | now: 250685.815644201, len(actionsLog): 0 2017-08-28 15:33:09,667 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:33:09,673 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:33:09,673 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:33:09,674 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 3.5333490909833927} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8490528172423086} master throughput: 0 total requests: 0 avg backup throughput: 0.8490528172423086 master throughput ratio: 0.0 2017-08-28 15:33:09,692 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:09,694 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:09,699 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:33:09,700 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:09,700 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 116 to run in 10 seconds 2017-08-28 15:33:09,717 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:09,719 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:09,723 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:09,735 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:09,738 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:09,738 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:09,741 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:09,741 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:09,742 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:09,742 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:09,784 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.134 seconds 2017-08-28 15:33:11,990 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:33:12,003 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:33:12,004 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:33:12,004 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:33:12,004 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:33:12,020 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:33:19,740 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 116 2017-08-28 15:33:19,740 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:33:19,740 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:19,753 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:33:19,753 | DEBUG | throttler.py ( 31) | acquire | now: 250695.90259035, len(actionsLog): 1 2017-08-28 15:33:19,754 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 1 2017-08-28 15:33:19,755 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.1025421927520482, timePassed: 10.086946148978313 2017-08-28 15:33:19,757 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 2 2017-08-28 15:33:19,757 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:33:19,758 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 3.5333490909833927} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8490528172423086} master throughput: 0 total requests: 0 avg backup throughput: 0.8490528172423086 master throughput ratio: 0.0 2017-08-28 15:33:19,777 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:19,778 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:19,779 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:33:19,781 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:19,782 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 117 to run in 10 seconds 2017-08-28 15:33:19,797 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:19,798 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:19,804 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:19,804 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:19,804 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:19,805 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:19,815 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:19,816 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:19,820 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:19,821 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:19,850 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.124 seconds 2017-08-28 15:33:26,923 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:33:26,943 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on request 1503934406877168 2017-08-28 15:33:26,945 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:26,953 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:26,962 | DEBUG | node.py (1690) | processRequest | Node6 received client request: SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:33:26,972 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:26,973 | DEBUG | propagator.py ( 148) | propagate | Node6 propagating Rhx2qwaeiGyhU9vn4ynHSS request 1503934406877168 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:33:26,979 | DEBUG | node.py (2566) | send | Node6 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:26,985 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} to its replicas since not finalised 2017-08-28 15:33:26,993 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934406877168,"op":"REQACK","identifier":"Rhx2qwaeiGyhU9vn4ynHSS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:33:27,002 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:33:27,011 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node5 2017-08-28 15:33:27,015 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:33:27,025 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node4 2017-08-28 15:33:27,031 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:33:27,039 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node1 2017-08-28 15:33:27,040 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:33:27,050 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node3 2017-08-28 15:33:27,051 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:33:27,061 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj","reqId":1503934406877168,"operation":{"type":"0","data":{"client_port":9702,"node_ip":"10.0.0.6","node_port":9701,"client_ip":"10.0.0.6","alias":"Node5","services":[]},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"PROPAGATE"}' to Node2 2017-08-28 15:33:27,074 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:33:27,074 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:33:27,080 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:33:27,080 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:33:27,080 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:33:27,087 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:33:27,097 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 5 messages through listener 2017-08-28 15:33:27,106 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503934406877168 2017-08-28 15:33:27,117 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,121 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, 'op': 'PROPAGATE'}, 'Node4') 2017-08-28 15:33:27,122 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,134 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503934406877168 2017-08-28 15:33:27,134 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,145 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, 'op': 'PROPAGATE'}, 'Node3') 2017-08-28 15:33:27,145 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,147 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503934406877168 2017-08-28 15:33:27,159 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,159 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, 'op': 'PROPAGATE'}, 'Node2') 2017-08-28 15:33:27,167 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,178 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503934406877168 2017-08-28 15:33:27,178 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,178 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, 'op': 'PROPAGATE'}, 'Node1') 2017-08-28 15:33:27,188 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,197 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503934406877168 2017-08-28 15:33:27,205 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,213 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, 'op': 'PROPAGATE'}, 'Node5') 2017-08-28 15:33:27,213 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,223 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,231 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:27,232 | DEBUG | propagator.py ( 213) | forward | Node6 forwarding request ('Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168) to 0 replicas 2017-08-28 15:33:27,240 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,250 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:27,251 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} to its replicas since already forwarded 2017-08-28 15:33:27,259 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,260 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:27,270 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'services': [], 'client_ip': '10.0.0.6', 'alias': 'Node5', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} to its replicas since already forwarded 2017-08-28 15:33:27,271 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,282 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:27,294 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'services': [], 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'node_port': 9701}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} to its replicas since already forwarded 2017-08-28 15:33:27,305 | DEBUG | node.py (1738) | processPropagate | Node Node6 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:33:27,306 | TRACE | propagator.py ( 141) | propagate | Node6 already propagated SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} 2017-08-28 15:33:27,307 | DEBUG | propagator.py ( 245) | tryForwarding | Node6 not forwarding request SafeRequest: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'} to its replicas since already forwarded 2017-08-28 15:33:27,377 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.474 seconds 2017-08-28 15:33:27,422 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:33:27,435 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'ppTime': 1503934407, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:27,435 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'discarded': 1, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'ppSeqNo': 10, 'instId': 1, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'stateRootHash': None, 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node4') 2017-08-28 15:33:27,444 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'ppTime': 1503934407, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:27,447 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:1 received PRE-PREPARE(2, 10) from Node4:1 at 250703.59596593 2017-08-28 15:33:27,470 | DEBUG | replica.py ( 934) | doPrepare | Node6:1 Sending PREPARE(2, 10) at 250703.61936828 2017-08-28 15:33:27,471 | DEBUG | replica.py (2133) | send | Node6:1 sending Prepare 2017-08-28 15:33:27,471 | TRACE | replica.py (2134) | send | Node6:1 sending PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,477 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,477 | DEBUG | replica.py ( 612) | trackBatches | Node6:1 tracking batch for PREPREPARE{'discarded': 1, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168)], 'ppTime': 1503934407, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} with state root None 2017-08-28 15:33:27,478 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:1 processed incoming PRE-PREPARE(2, 10) 2017-08-28 15:33:27,479 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:33:27,493 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'ppTime': 1503934407, 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:27,494 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'instId': 0, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'discarded': 0, 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'viewNo': 2, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'op': 'PREPREPARE'}, 'Node3') 2017-08-28 15:33:27,501 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168]], 'ppTime': 1503934407, 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:27,522 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:27,532 | DEBUG | replica.py ( 821) | processPrePrepare | Node6:0 received PRE-PREPARE(2, 10) from Node3:0 at 250703.681569053 2017-08-28 15:33:27,533 | DEBUG | replica.py (1034) | validate_pre_prepare | Node6:0 state root before processing PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168)], 'ppTime': 1503934407, 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} is b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9', 67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ 2017-08-28 15:33:27,544 | WARNING | replica.py ( 655) | processReqDuringBatch | Node6:0 encountered exception UnauthorizedClientRequest('Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot update a node',) while processing Request: {'signature': '2XNEbhmLphnQFgkovKFqgpenPhZooc24XrcFxoneYGoMfuzePhDs4vcFjgPQX9bsuemfbP6kGSkLg3r94uDWzDRj', 'reqId': 1503934406877168, 'operation': {'type': '0', 'data': {'client_port': 9702, 'alias': 'Node5', 'node_port': 9701, 'client_ip': '10.0.0.6', 'node_ip': '10.0.0.6', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}, will reject 2017-08-28 15:33:27,545 | DEBUG | replica.py ( 934) | doPrepare | Node6:0 Sending PREPARE(2, 10) at 250703.694162517 2017-08-28 15:33:27,552 | DEBUG | replica.py (2133) | send | Node6:0 sending Prepare 2017-08-28 15:33:27,552 | TRACE | replica.py (2134) | send | Node6:0 sending PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,552 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,560 | DEBUG | replica.py ( 612) | trackBatches | Node6:0 tracking batch for PREPREPARE{'discarded': 0, 'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168)], 'ppTime': 1503934407, 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} with state root b'8\xb0\xff\x81\xf2\x91\xbc\xac]w\xf3g\xea{RB\x15[\xbf\xfa\xaf\xff#\xf4\x03W@S,\xf1.\xa9' 2017-08-28 15:33:27,561 | DEBUG | replica.py ( 840) | processPrePrepare | Node6:0 processed incoming PRE-PREPARE(2, 10) 2017-08-28 15:33:27,585 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node5 2017-08-28 15:33:27,586 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node5 2017-08-28 15:33:27,593 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node4 2017-08-28 15:33:27,593 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node4 2017-08-28 15:33:27,604 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node1 2017-08-28 15:33:27,605 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node1 2017-08-28 15:33:27,605 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node3 2017-08-28 15:33:27,617 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node3 2017-08-28 15:33:27,618 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node2 2017-08-28 15:33:27,618 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":null,"stateRootHash":null,"op":"PREPARE","instId":1,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node2 2017-08-28 15:33:27,636 | TRACE | zstack.py ( 752) | transmitThroughListener | Node6C transmitting b'{"reqId":1503934406877168,"reason":"client request invalid: UnauthorizedClientRequest(\'Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot update a node\',)","op":"REJECT","identifier":"Rhx2qwaeiGyhU9vn4ynHSS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:33:27,647 | DEBUG | node.py (2566) | send | Node6 sending message PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:27,649 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:33:27,666 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,670 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppSeqNo': 10, 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:33:27,671 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,679 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,681 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'viewNo': 2, 'ppSeqNo': 10, 'stateRootHash': None, 'op': 'PREPARE'}, 'Node3') 2017-08-28 15:33:27,681 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,682 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,695 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'stateRootHash': None, 'ppSeqNo': 10}, 'Node5') 2017-08-28 15:33:27,695 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,702 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,702 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppSeqNo': 10, 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node4') 2017-08-28 15:33:27,703 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,748 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node5 2017-08-28 15:33:27,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node5 2017-08-28 15:33:27,758 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node4 2017-08-28 15:33:27,758 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node4 2017-08-28 15:33:27,771 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node1 2017-08-28 15:33:27,772 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node1 2017-08-28 15:33:27,772 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node3 2017-08-28 15:33:27,780 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node3 2017-08-28 15:33:27,780 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node2 2017-08-28 15:33:27,781 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"digest":"77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08","txnRootHash":"67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ","stateRootHash":"4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv","op":"PREPARE","instId":0,"ppTime":1503934407,"viewNo":2,"ppSeqNo":10}' to Node2 2017-08-28 15:33:27,794 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 10) from Node4:0 2017-08-28 15:33:27,810 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,811 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,823 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 10) from Node2:1 2017-08-28 15:33:27,823 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,824 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,824 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 10) from Node3:1 2017-08-28 15:33:27,832 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,833 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,833 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 10) from Node5:1 2017-08-28 15:33:27,833 | DEBUG | replica.py ( 953) | doCommit | Node6:1 Sending COMMIT(2, 10) at 250703.982469157 2017-08-28 15:33:27,842 | DEBUG | replica.py (2133) | send | Node6:1 sending Commit 2017-08-28 15:33:27,843 | TRACE | replica.py (2134) | send | Node6:1 sending COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:27,851 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:27,851 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,852 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 3 messages through listener 2017-08-28 15:33:27,865 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,865 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503934407, 'instId': 1, 'ppSeqNo': 10, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:33:27,873 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,874 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,874 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppSeqNo': 10, 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'op': 'PREPARE'}, 'Node2') 2017-08-28 15:33:27,887 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,888 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,888 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'op': 'PREPARE', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ppSeqNo': 10}, 'Node5') 2017-08-28 15:33:27,895 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,922 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:27,923 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 10) from Node2:0 2017-08-28 15:33:27,929 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,930 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,937 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 10) from Node5:0 2017-08-28 15:33:27,938 | DEBUG | replica.py ( 953) | doCommit | Node6:0 Sending COMMIT(2, 10) at 250704.086769228 2017-08-28 15:33:27,938 | DEBUG | replica.py (2133) | send | Node6:0 sending Commit 2017-08-28 15:33:27,945 | TRACE | replica.py (2134) | send | Node6:0 sending COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:27,945 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:27,945 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,951 | DEBUG | replica.py ( 865) | processPrepare | Node6:1 received PREPARE(2, 10) from Node1:1 2017-08-28 15:33:27,952 | DEBUG | replica.py ( 917) | tryCommit | Node6:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 10, 'stateRootHash': None} 2017-08-28 15:33:27,952 | DEBUG | replica.py ( 876) | processPrepare | Node6:1 processed incoming PREPARE (2, 10) 2017-08-28 15:33:27,953 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 2 messages through listener 2017-08-28 15:33:27,968 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,968 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'ppTime': 1503934407, 'instId': 0, 'ppSeqNo': 10, 'op': 'PREPARE'}, 'Node1') 2017-08-28 15:33:27,978 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:27,979 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:27,983 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 1, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:33:27,983 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:27,999 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node5 2017-08-28 15:33:28,005 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node5 2017-08-28 15:33:28,005 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node4 2017-08-28 15:33:28,014 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node4 2017-08-28 15:33:28,014 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node1 2017-08-28 15:33:28,015 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node1 2017-08-28 15:33:28,021 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node3 2017-08-28 15:33:28,021 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node3 2017-08-28 15:33:28,022 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node2 2017-08-28 15:33:28,031 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":1,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node2 2017-08-28 15:33:28,033 | DEBUG | node.py (2566) | send | Node6 sending message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:28,047 | DEBUG | replica.py ( 865) | processPrepare | Node6:0 received PREPARE(2, 10) from Node1:0 2017-08-28 15:33:28,048 | DEBUG | replica.py ( 917) | tryCommit | Node6:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'viewNo': 2, 'digest': '77e502d87a5ac0d47523e40804f47a9bcca2626a6da34ef7cde961e4a7075d08', 'ppTime': 1503934407, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ', 'instId': 0, 'ppSeqNo': 10, 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv'} 2017-08-28 15:33:28,048 | DEBUG | replica.py ( 876) | processPrepare | Node6:0 processed incoming PREPARE (2, 10) 2017-08-28 15:33:28,054 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 10) from Node3:1 2017-08-28 15:33:28,054 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,054 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,066 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:33:28,075 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,076 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:33:28,076 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,076 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,083 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:33:28,084 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,084 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,093 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:33:28,094 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,094 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,099 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'viewNo': 2, 'instId': 0, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node3') 2017-08-28 15:33:28,100 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,119 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node5 2017-08-28 15:33:28,122 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node5 2017-08-28 15:33:28,122 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node4 2017-08-28 15:33:28,123 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node4 2017-08-28 15:33:28,131 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node1 2017-08-28 15:33:28,132 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node1 2017-08-28 15:33:28,140 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node3 2017-08-28 15:33:28,140 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node3 2017-08-28 15:33:28,147 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node2 2017-08-28 15:33:28,148 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"instId":0,"viewNo":2,"ppSeqNo":10,"op":"COMMIT"}' to Node2 2017-08-28 15:33:28,149 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 10) from Node3:0 2017-08-28 15:33:28,165 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,165 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,165 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 10) from Node4:1 2017-08-28 15:33:28,166 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,174 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,175 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 10) from Node2:1 2017-08-28 15:33:28,175 | DEBUG | replica.py ( 929) | tryOrder | Node6:1 cannot return request to node: no quorum (5): COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,175 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,183 | DEBUG | replica.py ( 894) | processCommit | Node6:1 received COMMIT(2, 10) from Node5:1 2017-08-28 15:33:28,189 | TRACE | replica.py ( 925) | tryOrder | Node6:1 returning request to node 2017-08-28 15:33:28,189 | DEBUG | replica.py (1437) | doOrder | Node6:1 ordering COMMIT (2, 10) 2017-08-28 15:33:28,189 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:1 set last ordered as (2, 10) 2017-08-28 15:33:28,190 | DEBUG | replica.py (2133) | send | Node6:1 sending Ordered 2017-08-28 15:33:28,198 | TRACE | replica.py (2134) | send | Node6:1 sending ORDERED{'viewNo': 2, 'ppSeqNo': 10, 'ppTime': 1503934407, 'instId': 1, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503934406877168)], 'stateRootHash': None, 'ledgerId': 0, 'txnRootHash': None} 2017-08-28 15:33:28,199 | DEBUG | replica.py (1469) | order_3pc_key | Node6:1 ordered request (2, 10) 2017-08-28 15:33:28,206 | DEBUG | replica.py ( 905) | processCommit | Node6:1 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,218 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 4 messages through listener 2017-08-28 15:33:28,226 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,226 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node4') 2017-08-28 15:33:28,227 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,234 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,235 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 1, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:33:28,235 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,246 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,247 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node2') 2017-08-28 15:33:28,254 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,254 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node5: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,262 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node5') 2017-08-28 15:33:28,262 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,307 | TRACE | node.py (1839) | processOrdered | Node6 got ordered requests from backup replica 1 2017-08-28 15:33:28,331 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 10) from Node4:0 2017-08-28 15:33:28,342 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,343 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,343 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 10) from Node2:0 2017-08-28 15:33:28,361 | DEBUG | replica.py ( 929) | tryOrder | Node6:0 cannot return request to node: no quorum (5): COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} commits where f is 1 2017-08-28 15:33:28,362 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,375 | DEBUG | replica.py ( 894) | processCommit | Node6:0 received COMMIT(2, 10) from Node5:0 2017-08-28 15:33:28,379 | TRACE | replica.py ( 925) | tryOrder | Node6:0 returning request to node 2017-08-28 15:33:28,388 | DEBUG | replica.py (1437) | doOrder | Node6:0 ordering COMMIT (2, 10) 2017-08-28 15:33:28,388 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node6:0 set last ordered as (2, 10) 2017-08-28 15:33:28,389 | DEBUG | replica.py (2133) | send | Node6:0 sending Ordered 2017-08-28 15:33:28,394 | TRACE | replica.py (2134) | send | Node6:0 sending ORDERED{'viewNo': 2, 'ppSeqNo': 10, 'ppTime': 1503934407, 'instId': 0, 'reqIdr': [], 'stateRootHash': '4pJNQZ9fekfYgv7aw1YRNPURiZDfcp6AGE69J8osPxdv', 'ledgerId': 0, 'txnRootHash': '67d13kYEkmxc7hE3dBQ9mkkHKFZaZ2bBnUNt7nxQZ8yQ'} 2017-08-28 15:33:28,395 | DEBUG | replica.py (1469) | order_3pc_key | Node6:0 ordered request (2, 10) 2017-08-28 15:33:28,395 | DEBUG | replica.py ( 905) | processCommit | Node6:0 processed incoming COMMIT(2, 10) 2017-08-28 15:33:28,404 | TRACE | message_processor.py ( 29) | discard | Node6:1 discarding message COMMIT{'instId': 1, 'viewNo': 2, 'ppSeqNo': 10} because already ordered 3 phase message 2017-08-28 15:33:28,413 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6 got 1 messages through listener 2017-08-28 15:33:28,424 | DEBUG | node.py (1239) | validateNodeMsg | Node6 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,435 | DEBUG | node.py (1201) | handleOneNodeMsg | Node6 msg validated ({'instId': 0, 'viewNo': 2, 'ppSeqNo': 10, 'op': 'COMMIT'}, 'Node1') 2017-08-28 15:33:28,435 | DEBUG | node.py (1268) | postToNodeInBox | Node6 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} 2017-08-28 15:33:28,449 | DEBUG | node.py (1828) | processOrdered | Node6 executing Ordered batch 2 10 of 0 requests 2017-08-28 15:33:28,462 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:33:28,479 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node6 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:33:28,481 | TRACE | message_processor.py ( 29) | discard | Node6:0 discarding message COMMIT{'instId': 0, 'viewNo': 2, 'ppSeqNo': 10} because already ordered 3 phase message 2017-08-28 15:33:28,516 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.097 seconds 2017-08-28 15:33:29,805 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 117 2017-08-28 15:33:29,814 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:33:29,815 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:29,822 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:33:29,823 | DEBUG | throttler.py ( 31) | acquire | now: 250705.972537004, len(actionsLog): 2 2017-08-28 15:33:29,825 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 2 2017-08-28 15:33:29,827 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.2103418361512954, timePassed: 10.069946654024534 2017-08-28 15:33:29,842 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 3 2017-08-28 15:33:29,843 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:33:29,850 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:33:29,854 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:29,855 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:29,872 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:33:29,872 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:29,872 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 118 to run in 10 seconds 2017-08-28 15:33:29,876 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:29,888 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:29,888 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:29,888 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:29,899 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:29,900 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:29,900 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:29,907 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:29,907 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:29,915 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:29,951 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.148 seconds 2017-08-28 15:33:39,884 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 118 2017-08-28 15:33:39,885 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:33:39,903 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:39,904 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:33:39,904 | DEBUG | throttler.py ( 31) | acquire | now: 250716.053230275, len(actionsLog): 3 2017-08-28 15:33:39,904 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 3 2017-08-28 15:33:39,905 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.323668485456566, timePassed: 10.08069327098201 2017-08-28 15:33:39,908 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 4 2017-08-28 15:33:39,909 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:33:39,909 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:33:39,916 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:39,917 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:39,933 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:33:39,933 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:39,934 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 119 to run in 10 seconds 2017-08-28 15:33:39,940 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:39,951 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:39,952 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:39,952 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:39,952 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:39,953 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:39,956 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:39,957 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:39,957 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:39,958 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:39,987 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.109 seconds 2017-08-28 15:33:42,081 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:33:42,082 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:33:42,083 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:33:42,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:33:42,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:33:42,100 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:33:49,934 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 119 2017-08-28 15:33:49,935 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:33:49,936 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:33:49,958 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:33:49,958 | DEBUG | throttler.py ( 31) | acquire | now: 250726.107102447, len(actionsLog): 4 2017-08-28 15:33:49,958 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 4 2017-08-28 15:33:49,959 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.4428055163203397, timePassed: 10.053872171993135 2017-08-28 15:33:49,959 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 5 2017-08-28 15:33:49,979 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:33:49,980 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:33:49,981 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:33:49,981 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:49,982 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:33:49,982 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:33:49,982 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 120 to run in 10 seconds 2017-08-28 15:33:49,988 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:49,988 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:33:49,999 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:50,003 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:33:50,006 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:50,007 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:33:50,022 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:50,022 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:33:50,031 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:50,033 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:33:50,050 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.118 seconds 2017-08-28 15:33:57,091 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:33:57,091 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:33:57,099 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:33:57,100 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:33:57,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:33:57,114 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:33:59,993 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 120 2017-08-28 15:34:00,002 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:00,013 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:00,014 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:00,014 | DEBUG | throttler.py ( 31) | acquire | now: 250736.163128876, len(actionsLog): 5 2017-08-28 15:34:00,017 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:00,018 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.056026429025223 2017-08-28 15:34:00,025 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:00,025 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:00,026 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:00,053 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:00,053 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:00,054 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:00,055 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:00,067 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 121 to run in 10 seconds 2017-08-28 15:34:00,073 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:00,078 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:00,078 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:00,078 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:00,081 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:00,096 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:00,096 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:00,096 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:00,097 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:00,097 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:00,144 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.153 seconds 2017-08-28 15:34:07,226 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 115 2017-08-28 15:34:07,228 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:34:07,233 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:07,233 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 122 to run in 60 seconds 2017-08-28 15:34:07,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 17 2017-08-28 15:34:07,235 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:34:07,239 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 18 to run in 60 seconds 2017-08-28 15:34:10,082 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 121 2017-08-28 15:34:10,084 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:10,087 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:10,092 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:10,093 | DEBUG | throttler.py ( 31) | acquire | now: 250746.241818406, len(actionsLog): 6 2017-08-28 15:34:10,093 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:10,093 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.078689529997064 2017-08-28 15:34:10,093 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:10,094 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:10,094 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:10,111 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:10,116 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:10,116 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:10,116 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:10,116 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 123 to run in 10 seconds 2017-08-28 15:34:10,132 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:10,133 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:10,141 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:10,141 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:10,148 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:10,148 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:10,155 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:10,155 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:10,156 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:10,156 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:10,198 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.132 seconds 2017-08-28 15:34:12,093 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:34:12,093 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:34:12,094 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:34:12,095 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:34:12,097 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:34:12,098 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:34:20,130 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 123 2017-08-28 15:34:20,139 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:20,139 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:20,145 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:20,150 | DEBUG | throttler.py ( 31) | acquire | now: 250756.299422657, len(actionsLog): 6 2017-08-28 15:34:20,151 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:20,151 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.057604250992881 2017-08-28 15:34:20,152 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:20,152 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:20,160 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:20,170 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:20,174 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:20,181 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:20,183 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:20,187 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 124 to run in 10 seconds 2017-08-28 15:34:20,188 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:20,189 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:20,203 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:20,204 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:20,207 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:20,207 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:20,211 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:20,212 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:20,213 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:20,227 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:20,256 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.128 seconds 2017-08-28 15:34:27,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:34:27,106 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:34:27,106 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:34:27,108 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:34:27,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:34:27,122 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:34:30,199 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 124 2017-08-28 15:34:30,200 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:30,200 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:30,201 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:30,201 | DEBUG | throttler.py ( 31) | acquire | now: 250766.350224571, len(actionsLog): 6 2017-08-28 15:34:30,215 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:30,216 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.050801914010663 2017-08-28 15:34:30,216 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:30,216 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:30,222 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:30,230 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:30,231 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:30,231 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:30,245 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:30,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 125 to run in 10 seconds 2017-08-28 15:34:30,256 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:30,257 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:30,266 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:30,266 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:30,270 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:30,270 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:30,275 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:30,276 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:30,283 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:30,289 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:30,328 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.132 seconds 2017-08-28 15:34:40,264 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 125 2017-08-28 15:34:40,267 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:40,269 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:40,271 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:40,273 | DEBUG | throttler.py ( 31) | acquire | now: 250776.422530448, len(actionsLog): 6 2017-08-28 15:34:40,274 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:40,274 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.072305876994506 2017-08-28 15:34:40,274 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:40,274 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:40,275 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:40,293 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:40,298 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:40,308 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:40,308 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:40,309 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 126 to run in 10 seconds 2017-08-28 15:34:40,315 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:40,316 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:40,324 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:40,324 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:40,325 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:40,335 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:40,338 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:40,338 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:40,346 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:40,347 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:40,367 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.115 seconds 2017-08-28 15:34:42,130 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:34:42,142 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:34:42,143 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:34:42,143 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:34:42,144 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:34:42,154 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:34:50,309 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 126 2017-08-28 15:34:50,309 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:34:50,309 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:34:50,310 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:34:50,311 | DEBUG | throttler.py ( 31) | acquire | now: 250786.460311047, len(actionsLog): 6 2017-08-28 15:34:50,311 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:34:50,312 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.037780598999234 2017-08-28 15:34:50,312 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:34:50,312 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:34:50,327 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:34:50,332 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:34:50,333 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:50,333 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:34:50,348 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:34:50,351 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 127 to run in 10 seconds 2017-08-28 15:34:50,356 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:50,359 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:34:50,363 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:50,364 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:34:50,367 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:50,367 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:34:50,368 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:50,368 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:34:50,369 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:50,387 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:34:50,414 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.117 seconds 2017-08-28 15:34:57,141 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:34:57,152 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:34:57,152 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:34:57,152 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:34:57,154 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:34:57,156 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:35:00,361 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 127 2017-08-28 15:35:00,366 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:00,367 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:00,373 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:00,375 | DEBUG | throttler.py ( 31) | acquire | now: 250796.523897, len(actionsLog): 6 2017-08-28 15:35:00,375 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:00,378 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.063585953001166 2017-08-28 15:35:00,379 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:00,382 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:00,383 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:00,401 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:00,403 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:00,404 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:00,409 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:00,409 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 128 to run in 10 seconds 2017-08-28 15:35:00,410 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:00,429 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:00,430 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:00,430 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:00,431 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:00,431 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:00,441 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:00,441 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:00,447 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:00,448 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:00,483 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.126 seconds 2017-08-28 15:35:07,244 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 122 2017-08-28 15:35:07,244 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:35:07,244 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:07,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 129 to run in 60 seconds 2017-08-28 15:35:07,250 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 18 2017-08-28 15:35:07,251 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:35:07,254 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 19 to run in 60 seconds 2017-08-28 15:35:10,421 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 128 2017-08-28 15:35:10,423 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:10,425 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:10,425 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:10,426 | DEBUG | throttler.py ( 31) | acquire | now: 250806.574868221, len(actionsLog): 6 2017-08-28 15:35:10,439 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:10,440 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.050971221004147 2017-08-28 15:35:10,440 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:10,440 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:10,443 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:10,455 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:10,466 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:10,466 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:10,467 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:10,467 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 130 to run in 10 seconds 2017-08-28 15:35:10,470 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:10,478 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:10,478 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:10,479 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:10,490 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:10,491 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:10,491 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:10,497 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:10,497 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:10,498 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:10,536 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.120 seconds 2017-08-28 15:35:12,146 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:35:12,158 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:35:12,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:35:12,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:35:12,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:35:12,163 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:35:20,470 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 130 2017-08-28 15:35:20,485 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:20,485 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:20,485 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:20,487 | DEBUG | throttler.py ( 31) | acquire | now: 250816.636288383, len(actionsLog): 6 2017-08-28 15:35:20,487 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:20,488 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.061420161975548 2017-08-28 15:35:20,489 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:20,491 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:20,506 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:20,512 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:20,512 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:20,514 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:20,514 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:20,514 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 131 to run in 10 seconds 2017-08-28 15:35:20,534 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:20,535 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:20,535 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:20,543 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:20,543 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:20,544 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:20,552 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:20,553 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:20,553 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:20,554 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:20,598 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.106 seconds 2017-08-28 15:35:27,163 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:35:27,170 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:35:27,171 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:35:27,171 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:35:27,171 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:35:27,189 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:35:30,535 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 131 2017-08-28 15:35:30,539 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:30,540 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:30,547 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:30,548 | DEBUG | throttler.py ( 31) | acquire | now: 250826.696777835, len(actionsLog): 6 2017-08-28 15:35:30,548 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:30,548 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.06048945200746 2017-08-28 15:35:30,548 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:30,549 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:30,549 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:30,549 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:30,549 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:30,549 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:30,549 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:30,550 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 132 to run in 10 seconds 2017-08-28 15:35:30,577 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:30,577 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:30,593 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:30,595 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:30,596 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:30,597 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:30,597 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:30,597 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:30,611 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:30,612 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:30,648 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.121 seconds 2017-08-28 15:35:40,569 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 132 2017-08-28 15:35:40,576 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:40,576 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:40,576 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:40,576 | DEBUG | throttler.py ( 31) | acquire | now: 250836.725603219, len(actionsLog): 6 2017-08-28 15:35:40,579 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:40,590 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.028825384011725 2017-08-28 15:35:40,591 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:40,591 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:40,594 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:40,594 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:40,603 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:40,616 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:40,616 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:40,617 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 133 to run in 10 seconds 2017-08-28 15:35:40,629 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:40,629 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:40,630 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:40,630 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:40,634 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:40,635 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:40,646 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:40,650 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:40,651 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:40,651 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:40,695 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.137 seconds 2017-08-28 15:35:42,173 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:35:42,180 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:35:42,181 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:35:42,181 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:35:42,182 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:35:42,183 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:35:50,630 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 133 2017-08-28 15:35:50,631 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:35:50,639 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:35:50,640 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:35:50,640 | DEBUG | throttler.py ( 31) | acquire | now: 250846.789108727, len(actionsLog): 6 2017-08-28 15:35:50,640 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:35:50,641 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.063505508005619 2017-08-28 15:35:50,642 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:35:50,642 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:35:50,643 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:35:50,668 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:35:50,668 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:50,669 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:35:50,669 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:35:50,670 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 134 to run in 10 seconds 2017-08-28 15:35:50,691 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:50,693 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:35:50,693 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:50,698 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:35:50,698 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:50,699 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:35:50,703 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:50,705 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:35:50,711 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:50,712 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:35:50,734 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.106 seconds 2017-08-28 15:35:57,179 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:35:57,187 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:35:57,188 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:35:57,188 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:35:57,188 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:35:57,189 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:36:00,679 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 134 2017-08-28 15:36:00,694 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:00,695 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:00,695 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:00,698 | DEBUG | throttler.py ( 31) | acquire | now: 250856.847407256, len(actionsLog): 6 2017-08-28 15:36:00,699 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:00,700 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.058298528980231 2017-08-28 15:36:00,700 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:00,714 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:00,719 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:00,720 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:00,723 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:00,725 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:00,725 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:00,728 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 135 to run in 10 seconds 2017-08-28 15:36:00,739 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:00,745 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:00,746 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:00,746 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:00,752 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:00,753 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:00,762 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:00,764 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:00,766 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:00,766 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:00,789 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.113 seconds 2017-08-28 15:36:07,250 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 129 2017-08-28 15:36:07,265 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:36:07,268 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:07,269 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 136 to run in 60 seconds 2017-08-28 15:36:07,269 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 19 2017-08-28 15:36:07,269 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:36:07,270 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 20 to run in 60 seconds 2017-08-28 15:36:10,742 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 135 2017-08-28 15:36:10,746 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:10,746 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:10,747 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:10,752 | DEBUG | throttler.py ( 31) | acquire | now: 250866.900944793, len(actionsLog): 6 2017-08-28 15:36:10,752 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:10,752 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.053537536994554 2017-08-28 15:36:10,753 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:10,753 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:10,753 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:10,754 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:10,754 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:10,754 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:10,754 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:10,754 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 137 to run in 10 seconds 2017-08-28 15:36:10,778 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:10,779 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:10,780 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:10,780 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:10,781 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:10,803 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:10,803 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:10,809 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:10,810 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:10,810 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:10,843 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.103 seconds 2017-08-28 15:36:12,197 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:36:12,199 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:36:12,200 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:36:12,200 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:36:12,205 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:36:12,218 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:36:20,765 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 137 2017-08-28 15:36:20,774 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:20,774 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:20,774 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:20,775 | DEBUG | throttler.py ( 31) | acquire | now: 250876.923794872, len(actionsLog): 6 2017-08-28 15:36:20,775 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:20,777 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.022850079025375 2017-08-28 15:36:20,783 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:20,784 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:20,784 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:20,801 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:20,802 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:20,802 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:20,804 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:20,805 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 138 to run in 10 seconds 2017-08-28 15:36:20,807 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:20,807 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:20,808 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:20,808 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:20,808 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:20,830 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:20,830 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:20,831 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:20,839 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:20,840 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:20,870 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.108 seconds 2017-08-28 15:36:27,212 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:36:27,216 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:36:27,217 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:36:27,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:36:27,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:36:27,223 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:36:30,816 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 138 2017-08-28 15:36:30,821 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:30,821 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:30,821 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:30,822 | DEBUG | throttler.py ( 31) | acquire | now: 250886.970799376, len(actionsLog): 6 2017-08-28 15:36:30,822 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:30,823 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.047004503983771 2017-08-28 15:36:30,827 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:30,827 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:30,827 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:30,828 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:30,847 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:30,850 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:30,850 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:30,851 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 139 to run in 10 seconds 2017-08-28 15:36:30,856 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:30,856 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:30,871 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:30,875 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:30,879 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:30,880 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:30,880 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:30,880 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:30,888 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:30,889 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:30,913 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.119 seconds 2017-08-28 15:36:40,867 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 139 2017-08-28 15:36:40,873 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:40,873 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:40,874 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:40,875 | DEBUG | throttler.py ( 31) | acquire | now: 250897.023961362, len(actionsLog): 6 2017-08-28 15:36:40,875 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:40,888 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.053161985997576 2017-08-28 15:36:40,888 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:40,888 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:40,889 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:40,896 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:40,898 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:40,901 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:40,902 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:40,903 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 140 to run in 10 seconds 2017-08-28 15:36:40,908 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:40,909 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:40,925 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:40,925 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:40,931 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:40,933 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:40,935 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:40,936 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:40,947 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:40,951 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:40,979 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.130 seconds 2017-08-28 15:36:42,240 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:36:42,241 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:36:42,241 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:36:42,241 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:36:42,242 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:36:42,244 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:36:50,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 140 2017-08-28 15:36:50,927 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:36:50,929 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:36:50,929 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:36:50,930 | DEBUG | throttler.py ( 31) | acquire | now: 250907.078869078, len(actionsLog): 6 2017-08-28 15:36:50,930 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:36:50,930 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.054907715995796 2017-08-28 15:36:50,930 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:36:50,951 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:36:50,951 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:36:50,952 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:36:50,953 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:50,954 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:36:50,954 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:36:50,957 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 141 to run in 10 seconds 2017-08-28 15:36:50,976 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:50,976 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:36:50,978 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:50,979 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:36:50,990 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:50,991 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:36:50,991 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:50,999 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:36:51,005 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:51,006 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:36:51,037 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.135 seconds 2017-08-28 15:36:57,250 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:36:57,259 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:36:57,264 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:36:57,268 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:36:57,268 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:36:57,280 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:37:00,968 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 141 2017-08-28 15:37:00,968 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:00,968 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:00,969 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:00,975 | DEBUG | throttler.py ( 31) | acquire | now: 250917.123795965, len(actionsLog): 6 2017-08-28 15:37:00,975 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:00,980 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.044926886999747 2017-08-28 15:37:00,980 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:00,987 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:00,988 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:00,988 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:00,988 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:00,989 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:00,989 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:00,989 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 142 to run in 10 seconds 2017-08-28 15:37:01,002 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:01,002 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:01,004 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:01,005 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:01,005 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:01,005 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:01,005 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:01,006 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:01,006 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:01,006 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:01,053 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.105 seconds 2017-08-28 15:37:07,273 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 136 2017-08-28 15:37:07,277 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:37:07,283 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:07,284 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 143 to run in 60 seconds 2017-08-28 15:37:07,284 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 20 2017-08-28 15:37:07,284 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:37:07,285 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 21 to run in 60 seconds 2017-08-28 15:37:11,030 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 142 2017-08-28 15:37:11,031 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:11,035 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:11,039 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:11,040 | DEBUG | throttler.py ( 31) | acquire | now: 250927.188730977, len(actionsLog): 6 2017-08-28 15:37:11,040 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:11,042 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.064935012022033 2017-08-28 15:37:11,051 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:11,052 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:11,052 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:11,066 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:11,068 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:11,068 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:11,070 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:11,070 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 144 to run in 10 seconds 2017-08-28 15:37:11,083 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:11,088 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:11,088 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:11,088 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:11,098 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:11,099 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:11,099 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:11,100 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:11,100 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:11,106 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:11,145 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.126 seconds 2017-08-28 15:37:12,251 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:37:12,267 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:37:12,269 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:37:12,269 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:37:12,269 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:37:12,270 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:37:21,087 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 144 2017-08-28 15:37:21,091 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:21,095 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:21,095 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:21,099 | DEBUG | throttler.py ( 31) | acquire | now: 250937.24775966, len(actionsLog): 6 2017-08-28 15:37:21,100 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:21,100 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.05902868299745 2017-08-28 15:37:21,100 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:21,101 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:21,101 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:21,123 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:21,124 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:21,124 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:21,127 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:21,127 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 145 to run in 10 seconds 2017-08-28 15:37:21,144 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:21,145 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:21,152 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:21,152 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:21,153 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:21,156 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:21,157 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:21,157 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:21,172 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:21,173 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:21,197 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.122 seconds 2017-08-28 15:37:27,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:37:27,282 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:37:27,282 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:37:27,283 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:37:27,295 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:37:27,296 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:37:31,142 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 145 2017-08-28 15:37:31,143 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:31,146 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:31,147 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:31,148 | DEBUG | throttler.py ( 31) | acquire | now: 250947.297254868, len(actionsLog): 6 2017-08-28 15:37:31,152 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:31,152 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.049495207989821 2017-08-28 15:37:31,152 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:31,155 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:31,155 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:31,176 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:31,181 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:31,181 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:31,183 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:31,183 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 146 to run in 10 seconds 2017-08-28 15:37:31,205 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:31,206 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:31,206 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:31,212 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:31,212 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:31,213 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:31,213 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:31,224 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:31,225 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:31,225 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:31,261 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.122 seconds 2017-08-28 15:37:41,194 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 146 2017-08-28 15:37:41,196 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:41,196 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:41,197 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:41,197 | DEBUG | throttler.py ( 31) | acquire | now: 250957.346146684, len(actionsLog): 6 2017-08-28 15:37:41,211 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:41,211 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.048891815997194 2017-08-28 15:37:41,222 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:41,222 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:41,223 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:41,226 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:41,228 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:41,228 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:41,228 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:41,229 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 147 to run in 10 seconds 2017-08-28 15:37:41,247 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:41,251 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:41,252 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:41,252 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:41,253 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:41,253 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:41,261 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:41,261 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:41,269 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:41,269 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:41,309 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.131 seconds 2017-08-28 15:37:42,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:37:42,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:37:42,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:37:42,301 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:37:42,301 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:37:42,318 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:37:51,262 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 147 2017-08-28 15:37:51,263 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:37:51,269 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:37:51,269 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:37:51,269 | DEBUG | throttler.py ( 31) | acquire | now: 250967.418573039, len(actionsLog): 6 2017-08-28 15:37:51,270 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:37:51,271 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.072426355007337 2017-08-28 15:37:51,271 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:37:51,287 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:37:51,290 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:37:51,291 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:37:51,292 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:51,302 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:37:51,302 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:37:51,303 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 148 to run in 10 seconds 2017-08-28 15:37:51,320 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:51,321 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:37:51,321 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:51,325 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:37:51,325 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:51,333 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:37:51,334 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:51,334 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:37:51,345 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:51,346 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:37:51,373 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.123 seconds 2017-08-28 15:37:57,295 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:37:57,298 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:37:57,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:37:57,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:37:57,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:37:57,314 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:38:01,340 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 148 2017-08-28 15:38:01,342 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:38:01,342 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:38:01,343 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:38:01,343 | DEBUG | throttler.py ( 31) | acquire | now: 250977.49240672, len(actionsLog): 6 2017-08-28 15:38:01,346 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:38:01,346 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.073833681002725 2017-08-28 15:38:01,346 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:38:01,347 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:38:01,347 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:38:01,368 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:38:01,369 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:01,371 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:38:01,372 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:01,374 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 149 to run in 10 seconds 2017-08-28 15:38:01,391 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:01,392 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:01,395 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:01,404 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:01,404 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:01,404 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:01,404 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:01,405 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:01,409 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:01,409 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:01,441 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.121 seconds 2017-08-28 15:38:07,303 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkNodeRequestSpike with id 143 2017-08-28 15:38:07,303 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node6 checking its request amount 2017-08-28 15:38:07,306 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:38:07,307 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkNodeRequestSpike with id 150 to run in 60 seconds 2017-08-28 15:38:07,326 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 21 2017-08-28 15:38:07,327 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:38:07,327 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 22 to run in 60 seconds 2017-08-28 15:38:11,394 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 149 2017-08-28 15:38:11,401 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:38:11,402 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:38:11,402 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:38:11,402 | DEBUG | throttler.py ( 31) | acquire | now: 250987.551579192, len(actionsLog): 6 2017-08-28 15:38:11,403 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:38:11,403 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.059172471985221 2017-08-28 15:38:11,403 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:38:11,403 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:38:11,404 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:38:11,428 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:38:11,429 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:11,429 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:38:11,429 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:11,429 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 151 to run in 10 seconds 2017-08-28 15:38:11,440 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:11,453 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:11,454 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:11,455 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:11,455 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:11,459 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:11,466 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:11,478 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:11,479 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:11,479 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:11,509 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.118 seconds 2017-08-28 15:38:12,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:38:12,309 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:38:12,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:38:12,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:38:12,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:38:12,311 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:38:21,435 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 151 2017-08-28 15:38:21,438 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:38:21,438 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:38:21,439 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:38:21,439 | DEBUG | throttler.py ( 31) | acquire | now: 250997.588064417, len(actionsLog): 6 2017-08-28 15:38:21,439 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:38:21,443 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.036485225020442 2017-08-28 15:38:21,444 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:38:21,444 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:38:21,459 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:38:21,464 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:38:21,466 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:21,469 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:38:21,470 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:21,471 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 152 to run in 10 seconds 2017-08-28 15:38:21,478 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:21,478 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:21,495 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:21,495 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:21,496 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:21,496 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:21,498 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:21,506 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:21,512 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:21,513 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:21,552 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.130 seconds 2017-08-28 15:38:27,323 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:38:27,323 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node5 HA(host='10.0.0.6', port=9701) 2017-08-28 15:38:27,324 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:38:27,324 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:38:27,324 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node6 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:38:27,331 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node6 next check for retries in 15.00 seconds 2017-08-28 15:38:31,479 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node6 running action checkPerformance with id 152 2017-08-28 15:38:31,482 | TRACE | node.py (1959) | checkPerformance | Node6 checking its performance 2017-08-28 15:38:31,483 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:38:31,484 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node6 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:38:31,484 | DEBUG | throttler.py ( 31) | acquire | now: 251007.633174024, len(actionsLog): 6 2017-08-28 15:38:31,484 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 5 2017-08-28 15:38:31,484 | DEBUG | throttler.py ( 44) | acquire | timeToWaitAfterPreviousTry: 3.568050833375483, timePassed: 10.045109606988262 2017-08-28 15:38:31,484 | DEBUG | throttler.py ( 49) | acquire | timeToWaitAfterPreviousTry < timePassed was true, after append, len(actionsLog): 6 2017-08-28 15:38:31,485 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node6 sending an instance change with view_no 3 since Primary of master protocol instance degraded the performance 2017-08-28 15:38:31,485 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node6 metrics for monitor: Node6 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249722.956898051, 249722.957477207] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 4.624991182965459} master request latencies: {} client avg request latencies: [{}, {'6XiqM9DwYDgFnmyQQBFr72': (1, 1.1802929400000721), 'Rhx2qwaeiGyhU9vn4ynHSS': (1, 1.0916420919820666), 'FBQ5W7LxRp8bLeVNEafjpd': (1, 1.1182073199888691), 'XhYtvJqezMUKfF6KVNaGmT': (1, 1.2348488309944514)}] throughput: {0: 0, 1: 0.8648665136341457} master throughput: 0 total requests: 0 avg backup throughput: 0.8648665136341457 master throughput ratio: 0.0 2017-08-28 15:38:31,485 | DEBUG | node.py (2566) | send | Node6 sending message INSTANCE_CHANGE{'viewNo': 3, 'reason': 25} to all recipients: ['Node1', 'Node5', 'Node3', 'Node2', 'Node4'] 2017-08-28 15:38:31,485 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:31,485 | INFO | node.py (1971) | checkPerformance | Node6 sent view change since performance degraded of master instance 2017-08-28 15:38:31,486 | DEBUG | node.py (1939) | do_view_change_if_possible | Node6 has no quorum for view 3 2017-08-28 15:38:31,486 | TRACE | has_action_queue.py ( 36) | _schedule | Node6 scheduling action checkPerformance with id 153 to run in 10 seconds 2017-08-28 15:38:31,519 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:31,534 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node5 2017-08-28 15:38:31,534 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:31,535 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node4 2017-08-28 15:38:31,540 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:31,544 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node1 2017-08-28 15:38:31,544 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:31,545 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node3 2017-08-28 15:38:31,551 | DEBUG | zstack.py ( 721) | transmit | Node6 transmitting message b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:31,552 | TRACE | batched.py ( 95) | flushOutBoxes | Node6 sending msg b'{"viewNo":3,"reason":25,"op":"INSTANCE_CHANGE"}' to Node2 2017-08-28 15:38:31,592 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.128 seconds 2017-08-28 15:38:36,171 | TRACE | zstack.py ( 477) | _receiveFromListener | Node6C got 1 messages through listener 2017-08-28 15:38:36,194 | DEBUG | node.py (2217) | verifySignature | Node6 authenticated 5qhSKMkpqui6taGLTecwqf signature on request 1503934716120212 2017-08-28 15:38:36,215 | TRACE | node.py (1370) | validateClientMsg | Node6C received CLIENT message: SafeRequest: {'signature': '4FWtAeYRYzkrdr6Uopc6umZKdhaUZyWvD1o3i7UkyNdJsQk37cedA96NJ16dm1WKD45NgGy7KMpVm7teDMtio6Wb', 'reqId': 1503934716120212, 'operation': {'type': '0', 'data': {'client_port': 9702, 'node_ip': '10.0.0.6', 'node_port': 9701, 'client_ip': '10.0.0.6', 'alias': 'Node5', 'services': []}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'identifier': '5qhSKMkpqui6taGLTecwqf'} 2017-08-28 15:38:36,216 | DEBUG | node.py (1416) | processClientInBox | Node6C processing b'9C6z!fc&