2017-10-11 17:17:33,616 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node5.log 2017-10-11 17:17:33,617 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-10-11 17:17:34,698 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-11 17:17:34,714 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-11 17:17:34,752 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-11 17:17:34,754 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-11 17:17:35,589 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-11 17:17:35,590 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-10-11 17:17:35,620 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:17:35,628 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-10-11 17:17:35,640 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.012478787917643785 seconds 2017-10-11 17:17:35,657 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node5 2017-10-11 17:17:35,663 | INFO | node.py (2420) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2017-10-11 17:17:35,689 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:17:35,696 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-10-11 17:17:35,709 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.013134715147316456 seconds 2017-10-11 17:17:35,712 | INFO | node.py (2420) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2017-10-11 17:17:35,758 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:17:35,759 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:17:35,759 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:17:35,759 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:17:35,759 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-11 17:17:35,760 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-11 17:17:35,760 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-11 17:17:35,761 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-11 17:17:35,761 | DEBUG | replica.py ( 313) | h | Node5:0 set watermarks as 0 300 2017-10-11 17:17:35,761 | DISPLAY | replicas.py ( 36) | grow | Node5 added replica Node5:0 to instance 0 (master) 2017-10-11 17:17:35,762 | DEBUG | replica.py ( 313) | h | Node5:1 set watermarks as 0 300 2017-10-11 17:17:35,762 | DISPLAY | replicas.py ( 36) | grow | Node5 added replica Node5:1 to instance 1 (backup) 2017-10-11 17:17:35,763 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-11 17:17:35,763 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-11 17:17:35,764 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-11 17:17:35,764 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-11 17:17:35,764 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-11 17:17:35,765 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-10-11 17:17:35,765 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-11 17:17:35,765 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-10-11 17:17:35,784 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:17:35,793 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-10-11 17:17:35,805 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.012616791296750307 seconds 2017-10-11 17:17:35,813 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node5 has no upgrade events 2017-10-11 17:17:35,813 | INFO | node.py (2420) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2017-10-11 17:17:35,813 | DEBUG | motor.py ( 34) | set_status | Node5 changing status from stopped to starting 2017-10-11 17:17:35,814 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:17:35,815 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:17:35,815 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:17:35,815 | DEBUG | zstack.py ( 319) | start | Node5 starting with restricted as True and reSetupAuth as True 2017-10-11 17:17:35,816 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-11 17:17:35,816 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:17:35,816 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node5/public_keys] 2017-10-11 17:17:35,818 | DEBUG | zstack.py ( 347) | open | Node5 will bind its listener at 9701 2017-10-11 17:17:35,819 | INFO | stacks.py ( 84) | start | CONNECTION: Node5 listening for other nodes at 0.0.0.0:9701 2017-10-11 17:17:35,820 | DEBUG | zstack.py ( 319) | start | Node5C starting with restricted as False and reSetupAuth as True 2017-10-11 17:17:35,820 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-11 17:17:35,820 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:17:35,821 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-11 17:17:35,821 | DEBUG | zstack.py ( 347) | open | Node5C will bind its listener at 9702 2017-10-11 17:17:35,822 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-10-11 17:17:35,822 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-10-11 17:17:35,823 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-10-11 17:17:35,823 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-10-11 17:17:35,823 | INFO | node.py ( 621) | start | Node5 first time running... 2017-10-11 17:17:35,825 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node5 found the following missing connections: Node1, Node3, Node4, Node2 2017-10-11 17:17:35,826 | TRACE | remote.py ( 86) | connect | connecting socket 78 43040992 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:17:35,827 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node1 at 10.0.0.2:9701 2017-10-11 17:17:35,828 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:17:35,829 | TRACE | remote.py ( 86) | connect | connecting socket 81 43063712 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:17:35,831 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node3 at 10.0.0.4:9705 2017-10-11 17:17:35,832 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:17:35,832 | TRACE | remote.py ( 86) | connect | connecting socket 83 43068336 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:17:35,833 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node4 at 10.0.0.5:9707 2017-10-11 17:17:35,833 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:17:35,833 | TRACE | remote.py ( 86) | connect | connecting socket 84 43220928 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:17:35,834 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node2 at 10.0.0.3:9703 2017-10-11 17:17:35,834 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:17:35,835 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:17:35,848 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:17:35,849 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:17:35,849 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:17:35,849 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:17:35,849 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:17:35,850 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:17:35,850 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:17:35,850 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:17:38,831 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 4 2017-10-11 17:17:45,772 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 1 2017-10-11 17:17:45,772 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:17:45,773 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-11 17:17:50,830 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:17:50,830 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:17:50,830 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:17:50,830 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:17:50,830 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:17:50,831 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:17:50,831 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:17:50,831 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:17:50,831 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:17:50,842 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:17:50,843 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:17:50,843 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:17:50,843 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:17:50,843 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:17:50,843 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:17:50,843 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:17:50,843 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:17:55,789 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 6 2017-10-11 17:17:55,789 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:17:55,789 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-10-11 17:18:05,794 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 7 2017-10-11 17:18:05,794 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:05,794 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-10-11 17:18:05,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:05,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:05,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:05,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:05,841 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:05,841 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:05,841 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43220928 2017-10-11 17:18:05,841 | TRACE | remote.py ( 86) | connect | connecting socket 87 43262688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:05,842 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:18:05,842 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:05,842 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:05,842 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43063712 2017-10-11 17:18:05,842 | TRACE | remote.py ( 86) | connect | connecting socket 84 43264336 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:05,843 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:18:05,843 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:05,843 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:05,843 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43068336 2017-10-11 17:18:05,843 | TRACE | remote.py ( 86) | connect | connecting socket 82 43254640 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:05,843 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:18:05,844 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:05,844 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:05,844 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43040992 2017-10-11 17:18:05,844 | TRACE | remote.py ( 86) | connect | connecting socket 86 43063232 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:05,844 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:18:05,844 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:18:05,855 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:18:05,856 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:18:05,856 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:18:05,856 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:18:05,856 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:18:05,856 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:18:05,856 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:18:05,856 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:18:15,797 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 8 2017-10-11 17:18:15,798 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:15,798 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-10-11 17:18:20,849 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:20,849 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:20,849 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:20,850 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:20,850 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:18:20,850 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:18:20,851 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:18:20,851 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:18:20,851 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:18:20,863 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:18:20,863 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:18:20,864 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:18:20,864 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:18:20,864 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:18:20,865 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:18:20,865 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:18:20,865 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:18:25,803 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 9 2017-10-11 17:18:25,804 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:25,804 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-10-11 17:18:35,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 2 2017-10-11 17:18:35,767 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:18:35,767 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds 2017-10-11 17:18:35,768 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 1 2017-10-11 17:18:35,768 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:18:35,768 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-10-11 17:18:35,814 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 10 2017-10-11 17:18:35,814 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:35,814 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-10-11 17:18:35,826 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 5 2017-10-11 17:18:35,827 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 13 to run in 60 seconds 2017-10-11 17:18:35,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action propose_view_change with id 3 2017-10-11 17:18:35,827 | DEBUG | throttler.py ( 31) | acquire | now: 2271401.2848833, len(actionsLog): 0 2017-10-11 17:18:35,827 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-10-11 17:18:35,827 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-10-11 17:18:35,827 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node5 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-10-11 17:18:35,828 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node5 metrics for monitor: Node5 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [2271341.219211642, 2271341.220000871] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-10-11 17:18:35,828 | DEBUG | node.py (2608) | send | Node5 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:18:35,828 | DEBUG | node.py (1958) | do_view_change_if_possible | Node5 has no quorum for view 1 2017-10-11 17:18:35,828 | INFO | node.py (2108) | propose_view_change | Node5 sent view change since was disconnected from primary for too long 2017-10-11 17:18:35,828 | DEBUG | node.py (1958) | do_view_change_if_possible | Node5 has no quorum for view 1 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node2 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:18:35,830 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node2 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node3 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:18:35,830 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node3 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node4 2017-10-11 17:18:35,830 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:18:35,830 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node4 2017-10-11 17:18:35,831 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node1 2017-10-11 17:18:35,831 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:18:35,831 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":26,"viewNo":1}' to Node1 2017-10-11 17:18:35,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:35,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:35,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:35,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:35,855 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:35,856 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:35,856 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43262688 2017-10-11 17:18:35,856 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138608 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:35,856 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:18:35,856 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:35,857 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:35,857 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43264336 2017-10-11 17:18:35,857 | TRACE | remote.py ( 86) | connect | connecting socket 83 43262688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:35,857 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:18:35,857 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:35,857 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:35,858 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43254640 2017-10-11 17:18:35,858 | TRACE | remote.py ( 86) | connect | connecting socket 81 43264336 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:35,858 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:18:35,858 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:35,858 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:35,858 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43063232 2017-10-11 17:18:35,859 | TRACE | remote.py ( 86) | connect | connecting socket 84 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:35,859 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:18:35,859 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:18:35,871 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:18:35,871 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:18:35,871 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:18:35,871 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:18:35,872 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:18:35,872 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:18:35,872 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:18:35,872 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:18:45,823 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 12 2017-10-11 17:18:45,824 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:45,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-10-11 17:18:50,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:18:50,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:18:50,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:18:50,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:18:50,856 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:18:50,857 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:18:50,857 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:18:50,857 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:18:50,857 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:18:50,869 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:18:50,869 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:18:50,870 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:18:50,870 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:18:50,870 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:18:50,871 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:18:50,871 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:18:50,871 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:18:55,833 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 14 2017-10-11 17:18:55,833 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:18:55,833 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-10-11 17:19:05,842 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 15 2017-10-11 17:19:05,843 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:05,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-10-11 17:19:05,857 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:05,857 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:05,858 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:05,858 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:05,858 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:05,859 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:05,859 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138608 2017-10-11 17:19:05,861 | TRACE | remote.py ( 86) | connect | connecting socket 82 43063776 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:05,863 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:19:05,864 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:05,864 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:05,864 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43262688 2017-10-11 17:19:05,865 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138608 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:05,865 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:19:05,867 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:05,867 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:05,867 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43264336 2017-10-11 17:19:05,867 | TRACE | remote.py ( 86) | connect | connecting socket 83 43262688 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:05,868 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:19:05,868 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:05,868 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:05,868 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43057408 2017-10-11 17:19:05,868 | TRACE | remote.py ( 86) | connect | connecting socket 85 43253520 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:05,868 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:19:05,868 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:19:05,880 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:19:05,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:19:05,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:19:05,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:19:05,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:19:05,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:19:05,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:19:05,882 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:19:15,846 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 16 2017-10-11 17:19:15,846 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:15,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-10-11 17:19:20,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:20,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:20,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:20,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:20,869 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:19:20,869 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:19:20,869 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:19:20,869 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:19:20,869 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:19:20,880 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:19:20,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:19:20,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:19:20,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:19:20,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:19:20,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:19:20,881 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:19:20,881 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:19:25,850 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 17 2017-10-11 17:19:25,850 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:25,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-10-11 17:19:35,776 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 11 2017-10-11 17:19:35,776 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:19:35,776 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds 2017-10-11 17:19:35,777 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 2 2017-10-11 17:19:35,777 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:19:35,777 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-10-11 17:19:35,835 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 13 2017-10-11 17:19:35,836 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 20 to run in 60 seconds 2017-10-11 17:19:35,860 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 18 2017-10-11 17:19:35,861 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:35,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-10-11 17:19:35,873 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:35,874 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:35,874 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:35,874 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:35,875 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:35,875 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:35,875 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43063776 2017-10-11 17:19:35,876 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:35,877 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:19:35,877 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:35,877 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:35,878 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138608 2017-10-11 17:19:35,878 | TRACE | remote.py ( 86) | connect | connecting socket 82 43063776 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:35,878 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:19:35,878 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:35,879 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:35,879 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43262688 2017-10-11 17:19:35,880 | TRACE | remote.py ( 86) | connect | connecting socket 84 43227120 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:35,880 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:19:35,880 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:35,880 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:35,881 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43253520 2017-10-11 17:19:35,881 | TRACE | remote.py ( 86) | connect | connecting socket 78 43262688 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:35,881 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:19:35,882 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:19:35,896 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:19:35,897 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:19:35,897 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:19:35,897 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:19:35,898 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:19:35,898 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:19:35,898 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:19:35,898 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:19:45,868 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 21 2017-10-11 17:19:45,869 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:45,869 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-10-11 17:19:50,881 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:19:50,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:19:50,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:19:50,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:19:50,883 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:19:50,883 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:19:50,883 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:19:50,884 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:19:50,884 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:19:50,897 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:19:50,897 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:19:50,898 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:19:50,898 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:19:50,899 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:19:50,899 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:19:50,899 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:19:50,900 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:19:55,877 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 22 2017-10-11 17:19:55,877 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:19:55,878 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-10-11 17:20:05,884 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 23 2017-10-11 17:20:05,884 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:05,884 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-10-11 17:20:05,884 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:05,884 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:05,884 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:05,884 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:05,885 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:05,885 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:05,885 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:20:05,885 | TRACE | remote.py ( 86) | connect | connecting socket 85 43138608 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:05,887 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:20:05,887 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:05,887 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:05,888 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43063776 2017-10-11 17:20:05,888 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:05,888 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:20:05,891 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:05,891 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:05,891 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43227120 2017-10-11 17:20:05,892 | TRACE | remote.py ( 86) | connect | connecting socket 89 43245184 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:05,892 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:20:05,892 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:05,892 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:05,892 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43262688 2017-10-11 17:20:05,892 | TRACE | remote.py ( 86) | connect | connecting socket 90 43247952 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:05,893 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:20:05,893 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:20:05,895 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:20:05,895 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:20:05,895 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:20:05,895 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:20:05,895 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:20:05,895 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:20:05,895 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:20:05,895 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:20:15,889 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 24 2017-10-11 17:20:15,889 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:15,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-10-11 17:20:20,886 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:20,890 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:20,891 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:20,891 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:20,891 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:20:20,891 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:20:20,891 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:20:20,891 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:20:20,891 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:20:20,903 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:20:20,904 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:20:20,904 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:20:20,904 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:20:20,905 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:20:20,905 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:20:20,905 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:20:20,905 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:20:25,896 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 25 2017-10-11 17:20:25,896 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:25,896 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-10-11 17:20:35,779 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 19 2017-10-11 17:20:35,779 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:20:35,780 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds 2017-10-11 17:20:35,780 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 3 2017-10-11 17:20:35,780 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:20:35,780 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-10-11 17:20:35,837 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 20 2017-10-11 17:20:35,837 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 28 to run in 60 seconds 2017-10-11 17:20:35,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:35,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:35,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:35,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:35,888 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:35,889 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:35,889 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43138608 2017-10-11 17:20:35,889 | TRACE | remote.py ( 86) | connect | connecting socket 82 43262688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:35,889 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:20:35,889 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:35,891 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:35,892 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:20:35,892 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138608 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:35,893 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:20:35,893 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:35,893 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:35,893 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43245184 2017-10-11 17:20:35,893 | TRACE | remote.py ( 86) | connect | connecting socket 84 43063776 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:35,894 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:20:35,894 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:35,894 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:35,894 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 90 43247952 2017-10-11 17:20:35,894 | TRACE | remote.py ( 86) | connect | connecting socket 88 43243536 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:35,894 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:20:35,895 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:20:35,906 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 26 2017-10-11 17:20:35,906 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:35,906 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-10-11 17:20:35,906 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:20:35,906 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:20:35,907 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:20:35,907 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:20:35,907 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:20:35,907 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:20:35,907 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:20:35,907 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:20:45,909 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 29 2017-10-11 17:20:45,911 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:45,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-10-11 17:20:50,889 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:20:50,890 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:20:50,890 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:20:50,890 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:20:50,890 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:20:50,890 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:20:50,890 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:20:50,891 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:20:50,891 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:20:50,902 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:20:50,903 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:20:50,903 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:20:50,903 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:20:50,903 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:20:50,903 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:20:50,903 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:20:50,903 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:20:55,918 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 30 2017-10-11 17:20:55,918 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:20:55,919 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-10-11 17:21:05,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:05,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:05,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:05,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:05,900 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:05,901 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:05,901 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43262688 2017-10-11 17:21:05,901 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:05,901 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:21:05,901 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:05,901 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:05,902 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138608 2017-10-11 17:21:05,902 | TRACE | remote.py ( 86) | connect | connecting socket 82 43262688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:05,902 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:21:05,902 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:05,902 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:05,902 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43063776 2017-10-11 17:21:05,902 | TRACE | remote.py ( 86) | connect | connecting socket 83 43252352 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:05,903 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:21:05,903 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:05,903 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:05,903 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43243536 2017-10-11 17:21:05,903 | TRACE | remote.py ( 86) | connect | connecting socket 89 43255120 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:05,903 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:21:05,903 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:21:05,915 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:21:05,915 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:21:05,915 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:21:05,916 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:21:05,916 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:21:05,916 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:21:05,916 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:21:05,916 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:21:05,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 31 2017-10-11 17:21:05,928 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:05,928 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-10-11 17:21:15,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 32 2017-10-11 17:21:15,936 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:15,937 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-10-11 17:21:20,905 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:20,905 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:20,905 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:20,905 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:20,905 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:21:20,905 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:21:20,905 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:21:20,906 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:21:20,906 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:21:20,917 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:21:20,917 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:21:20,918 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:21:20,918 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:21:20,918 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:21:20,918 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:21:20,918 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:21:20,918 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:21:25,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 33 2017-10-11 17:21:25,947 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:25,948 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-10-11 17:21:35,780 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 27 2017-10-11 17:21:35,781 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:21:35,781 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds 2017-10-11 17:21:35,781 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 4 2017-10-11 17:21:35,781 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:21:35,781 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-10-11 17:21:35,839 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 28 2017-10-11 17:21:35,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 36 to run in 60 seconds 2017-10-11 17:21:35,907 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:35,908 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:35,908 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:35,908 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:35,908 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:35,908 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:35,908 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:21:35,909 | TRACE | remote.py ( 86) | connect | connecting socket 78 43063776 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:35,909 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:21:35,909 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:35,909 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:35,909 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43262688 2017-10-11 17:21:35,910 | TRACE | remote.py ( 86) | connect | connecting socket 81 43138608 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:35,910 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:21:35,910 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:35,910 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:35,910 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43252352 2017-10-11 17:21:35,911 | TRACE | remote.py ( 86) | connect | connecting socket 85 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:35,911 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:21:35,911 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:35,911 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:35,911 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43255120 2017-10-11 17:21:35,911 | TRACE | remote.py ( 86) | connect | connecting socket 83 43252352 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:35,911 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:21:35,912 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:21:35,924 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:21:35,924 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:21:35,924 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:21:35,924 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:21:35,925 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:21:35,925 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:21:35,925 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:21:35,925 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:21:35,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 34 2017-10-11 17:21:35,951 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:35,951 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 37 to run in 10 seconds 2017-10-11 17:21:45,958 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 37 2017-10-11 17:21:45,958 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:45,958 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-10-11 17:21:50,911 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:21:50,911 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:21:50,911 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:21:50,911 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:21:50,911 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:21:50,911 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:21:50,912 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:21:50,912 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:21:50,912 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:21:50,923 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:21:50,924 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:21:50,924 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:21:50,924 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:21:50,924 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:21:50,924 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:21:50,925 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:21:50,925 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:21:55,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 38 2017-10-11 17:21:55,971 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:21:55,971 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-10-11 17:22:05,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:05,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:05,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:05,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:05,921 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:05,922 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:05,922 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43063776 2017-10-11 17:22:05,922 | TRACE | remote.py ( 86) | connect | connecting socket 87 43255120 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:05,922 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:22:05,922 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:05,922 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:05,922 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43138608 2017-10-11 17:22:05,923 | TRACE | remote.py ( 86) | connect | connecting socket 89 43263808 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:05,923 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:22:05,923 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:05,923 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:05,923 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43057408 2017-10-11 17:22:05,923 | TRACE | remote.py ( 86) | connect | connecting socket 90 43244656 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:05,924 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:22:05,924 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:05,924 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:05,924 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43252352 2017-10-11 17:22:05,924 | TRACE | remote.py ( 86) | connect | connecting socket 91 43247664 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:05,924 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:22:05,924 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:22:05,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:22:05,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:22:05,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:22:05,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:22:05,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:22:05,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:22:05,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:22:05,937 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:22:05,982 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 39 2017-10-11 17:22:05,982 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:05,982 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-10-11 17:22:15,985 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 40 2017-10-11 17:22:15,985 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:15,985 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-10-11 17:22:20,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:20,922 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:20,922 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:20,922 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:20,922 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:22:20,923 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:22:20,923 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:22:20,923 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:22:20,923 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:22:20,935 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:22:20,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:22:20,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:22:20,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:22:20,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:22:20,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:22:20,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:22:20,936 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:22:25,986 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 41 2017-10-11 17:22:25,986 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:25,987 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-10-11 17:22:35,785 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 35 2017-10-11 17:22:35,786 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:22:35,786 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 43 to run in 60 seconds 2017-10-11 17:22:35,786 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 5 2017-10-11 17:22:35,787 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:22:35,787 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-10-11 17:22:35,844 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 36 2017-10-11 17:22:35,845 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 44 to run in 60 seconds 2017-10-11 17:22:35,926 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:35,927 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:35,927 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:35,927 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:35,927 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:35,928 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:35,928 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43255120 2017-10-11 17:22:35,928 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138848 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:35,929 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:22:35,929 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:35,929 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:35,930 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43263808 2017-10-11 17:22:35,930 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:35,930 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:22:35,930 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:35,931 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:35,931 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 90 43244656 2017-10-11 17:22:35,931 | TRACE | remote.py ( 86) | connect | connecting socket 82 43063776 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:35,932 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:22:35,932 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:35,932 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:35,932 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 91 43247664 2017-10-11 17:22:35,933 | TRACE | remote.py ( 86) | connect | connecting socket 85 43263808 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:35,934 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:22:35,935 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:22:35,948 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:22:35,948 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:22:35,949 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:22:35,949 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:22:35,949 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:22:35,949 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:22:35,950 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:22:35,950 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:22:35,996 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 42 2017-10-11 17:22:35,996 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:35,996 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 45 to run in 10 seconds 2017-10-11 17:22:46,000 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 45 2017-10-11 17:22:46,001 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:46,001 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-10-11 17:22:50,927 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:22:50,928 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:22:50,928 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:22:50,928 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:22:50,928 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:22:50,928 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:22:50,928 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:22:50,929 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:22:50,929 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:22:50,940 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:22:50,941 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:22:50,941 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:22:50,941 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:22:50,941 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:22:50,941 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:22:50,942 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:22:50,942 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:22:56,010 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 46 2017-10-11 17:22:56,010 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:22:56,010 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-10-11 17:23:05,928 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:05,928 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:05,929 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:05,929 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:05,929 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:05,929 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:05,929 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138848 2017-10-11 17:23:05,930 | TRACE | remote.py ( 86) | connect | connecting socket 87 43253360 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:05,930 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:23:05,930 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:05,931 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:05,931 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:23:05,931 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138848 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:05,931 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:23:05,931 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:05,931 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:05,932 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43063776 2017-10-11 17:23:05,932 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:05,932 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:23:05,932 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:05,932 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:05,932 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43263808 2017-10-11 17:23:05,940 | TRACE | remote.py ( 86) | connect | connecting socket 83 43263808 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:05,940 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:23:05,940 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:23:05,952 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:23:05,952 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:23:05,952 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:23:05,952 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:23:05,952 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:23:05,953 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:23:05,953 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:23:05,953 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:23:06,011 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 47 2017-10-11 17:23:06,011 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:06,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-10-11 17:23:16,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 48 2017-10-11 17:23:16,024 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:16,024 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-10-11 17:23:20,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:20,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:20,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:20,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:20,931 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:23:20,931 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:23:20,931 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:23:20,931 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:23:20,932 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:23:20,943 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:23:20,944 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:23:20,944 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:23:20,944 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:23:20,945 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:23:20,945 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:23:20,945 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:23:20,946 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:23:26,036 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 49 2017-10-11 17:23:26,036 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:26,036 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-10-11 17:23:35,791 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 43 2017-10-11 17:23:35,792 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:23:35,792 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 51 to run in 60 seconds 2017-10-11 17:23:35,792 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 6 2017-10-11 17:23:35,792 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:23:35,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-10-11 17:23:35,852 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 44 2017-10-11 17:23:35,853 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 52 to run in 60 seconds 2017-10-11 17:23:35,935 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:35,935 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:35,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:35,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:35,936 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:35,936 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:35,936 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43253360 2017-10-11 17:23:35,937 | TRACE | remote.py ( 86) | connect | connecting socket 85 43063776 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:35,942 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:23:35,942 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:35,942 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:35,943 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138848 2017-10-11 17:23:35,943 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138848 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:35,944 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:23:35,944 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:35,944 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:35,944 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:23:35,945 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:35,947 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:23:35,947 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:35,947 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:35,947 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43263808 2017-10-11 17:23:35,948 | TRACE | remote.py ( 86) | connect | connecting socket 87 43253360 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:35,948 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:23:35,948 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:23:35,960 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:23:35,960 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:23:35,960 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:23:35,960 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:23:35,960 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:23:35,960 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:23:35,960 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:23:35,961 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:23:36,042 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 50 2017-10-11 17:23:36,043 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:36,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-10-11 17:23:46,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 53 2017-10-11 17:23:46,043 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:46,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-10-11 17:23:50,941 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:23:50,941 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:23:50,941 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:23:50,941 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:23:50,941 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:23:50,941 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:23:50,942 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:23:50,942 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:23:50,943 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:23:50,955 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:23:50,955 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:23:50,956 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:23:50,956 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:23:50,956 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:23:50,957 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:23:50,957 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:23:50,957 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:23:56,048 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 54 2017-10-11 17:23:56,048 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:23:56,048 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-10-11 17:24:05,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:05,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:05,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:05,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:05,948 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:05,949 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:05,949 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43063776 2017-10-11 17:24:05,949 | TRACE | remote.py ( 86) | connect | connecting socket 84 43263808 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:05,949 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:24:05,950 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:05,950 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:05,950 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138848 2017-10-11 17:24:05,951 | TRACE | remote.py ( 86) | connect | connecting socket 85 43063776 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:05,951 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:24:05,951 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:05,951 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:05,952 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:24:05,953 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138848 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:05,953 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:24:05,953 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:05,954 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:05,954 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43253360 2017-10-11 17:24:05,954 | TRACE | remote.py ( 86) | connect | connecting socket 82 43040992 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:05,954 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:24:05,954 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:24:05,965 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:24:05,966 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:24:05,967 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:24:05,967 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:24:05,967 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:24:05,967 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:24:05,967 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:24:05,967 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:24:06,058 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 55 2017-10-11 17:24:06,058 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:06,058 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-10-11 17:24:16,068 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 56 2017-10-11 17:24:16,069 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:16,069 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-10-11 17:24:20,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:20,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:20,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:20,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:20,953 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:24:20,953 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:24:20,953 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:24:20,953 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:24:20,953 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:24:20,964 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:24:20,965 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:24:20,965 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:24:20,965 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:24:20,965 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:24:20,965 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:24:20,965 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:24:20,965 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:24:26,072 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 57 2017-10-11 17:24:26,072 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:26,072 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-10-11 17:24:35,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 51 2017-10-11 17:24:35,800 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:24:35,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds 2017-10-11 17:24:35,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 7 2017-10-11 17:24:35,801 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:24:35,801 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-10-11 17:24:35,857 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 52 2017-10-11 17:24:35,858 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 60 to run in 60 seconds 2017-10-11 17:24:35,954 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:35,955 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:35,955 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:35,956 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:35,956 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:35,957 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:35,957 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43263808 2017-10-11 17:24:35,958 | TRACE | remote.py ( 86) | connect | connecting socket 87 43253360 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:35,958 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:24:35,959 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:35,959 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:35,959 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43063776 2017-10-11 17:24:35,960 | TRACE | remote.py ( 86) | connect | connecting socket 84 43263808 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:35,960 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:24:35,961 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:35,963 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:35,963 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138848 2017-10-11 17:24:35,964 | TRACE | remote.py ( 86) | connect | connecting socket 78 43063776 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:35,964 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:24:35,964 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:35,965 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:35,965 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43040992 2017-10-11 17:24:35,966 | TRACE | remote.py ( 86) | connect | connecting socket 85 43243536 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:35,966 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:24:35,966 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:24:35,978 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:24:35,978 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:24:35,980 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:24:35,980 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:24:35,982 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:24:35,982 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:24:35,982 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:24:35,982 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:24:36,077 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 58 2017-10-11 17:24:36,077 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:36,077 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 61 to run in 10 seconds 2017-10-11 17:24:46,082 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 61 2017-10-11 17:24:46,083 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:46,083 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-10-11 17:24:50,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:24:50,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:24:50,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:24:50,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:24:50,957 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:24:50,958 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:24:50,958 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:24:50,958 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:24:50,958 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:24:50,970 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:24:50,970 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:24:50,970 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:24:50,971 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:24:50,971 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:24:50,971 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:24:50,971 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:24:50,971 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:24:56,085 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 62 2017-10-11 17:24:56,085 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:24:56,086 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-10-11 17:25:05,968 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:05,968 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:05,968 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:05,968 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:05,968 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:05,969 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:05,969 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43253360 2017-10-11 17:25:05,969 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138848 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:05,970 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:25:05,970 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:05,970 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:05,970 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43263808 2017-10-11 17:25:05,972 | TRACE | remote.py ( 86) | connect | connecting socket 82 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:05,972 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:25:05,972 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:05,972 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:05,972 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43063776 2017-10-11 17:25:05,972 | TRACE | remote.py ( 86) | connect | connecting socket 83 43252240 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:05,973 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:25:05,973 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:05,973 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:05,973 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43243536 2017-10-11 17:25:05,973 | TRACE | remote.py ( 86) | connect | connecting socket 84 43262688 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:05,973 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:25:05,973 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:25:05,985 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:25:05,985 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:25:05,986 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:25:05,986 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:25:05,986 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:25:05,987 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:25:05,987 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:25:05,987 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:25:06,090 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 63 2017-10-11 17:25:06,090 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:06,090 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-10-11 17:25:16,097 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 64 2017-10-11 17:25:16,097 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:16,097 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-10-11 17:25:20,974 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:20,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:20,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:20,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:20,975 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:25:20,976 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:25:20,976 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:25:20,976 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:25:20,976 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:25:20,989 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:25:20,993 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:25:20,994 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:25:20,994 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:25:20,994 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:25:20,994 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:25:20,995 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:25:20,995 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:25:26,097 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 65 2017-10-11 17:25:26,098 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:26,098 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 66 to run in 10 seconds 2017-10-11 17:25:35,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 59 2017-10-11 17:25:35,811 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:25:35,812 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 67 to run in 60 seconds 2017-10-11 17:25:35,812 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 8 2017-10-11 17:25:35,812 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:25:35,812 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-10-11 17:25:35,870 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 60 2017-10-11 17:25:35,871 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 68 to run in 60 seconds 2017-10-11 17:25:35,985 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:35,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:35,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:35,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:35,986 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:35,986 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:35,987 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138848 2017-10-11 17:25:35,987 | TRACE | remote.py ( 86) | connect | connecting socket 85 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:35,988 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:25:35,988 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:35,989 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:35,989 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43057408 2017-10-11 17:25:35,989 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138848 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:35,989 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:25:35,989 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:35,989 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:35,989 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43252240 2017-10-11 17:25:35,990 | TRACE | remote.py ( 86) | connect | connecting socket 81 43244656 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:35,990 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:25:35,990 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:35,990 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:35,990 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43262688 2017-10-11 17:25:35,992 | TRACE | remote.py ( 86) | connect | connecting socket 89 43247424 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:35,994 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:25:35,994 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:25:36,007 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:25:36,007 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:25:36,007 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:25:36,007 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:25:36,008 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:25:36,008 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:25:36,008 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:25:36,008 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:25:36,100 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 66 2017-10-11 17:25:36,100 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:36,100 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-10-11 17:25:46,102 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 69 2017-10-11 17:25:46,103 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:46,103 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-10-11 17:25:50,989 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:25:50,989 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:25:50,989 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:25:50,989 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:25:50,990 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:25:50,990 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:25:50,990 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:25:50,990 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:25:50,990 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:25:51,002 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:25:51,002 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:25:51,002 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:25:51,002 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:25:51,002 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:25:51,003 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:25:51,003 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:25:51,003 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:25:56,104 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 70 2017-10-11 17:25:56,104 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:25:56,105 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-10-11 17:26:05,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:05,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:05,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:05,992 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:05,992 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:05,993 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:05,993 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43064688 2017-10-11 17:26:05,994 | TRACE | remote.py ( 86) | connect | connecting socket 82 43057408 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:05,994 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:26:05,995 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:05,995 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:05,995 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138848 2017-10-11 17:26:05,995 | TRACE | remote.py ( 86) | connect | connecting socket 84 43064688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:05,995 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:26:05,995 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:05,995 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:05,996 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43244656 2017-10-11 17:26:05,996 | TRACE | remote.py ( 86) | connect | connecting socket 85 43252240 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:05,996 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:26:05,996 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:05,996 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:05,997 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43247424 2017-10-11 17:26:05,997 | TRACE | remote.py ( 86) | connect | connecting socket 86 43262688 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:05,997 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:26:05,997 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:26:06,010 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:26:06,010 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:26:06,010 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:26:06,010 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:26:06,010 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:26:06,010 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:26:06,011 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:26:06,012 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:26:06,105 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 71 2017-10-11 17:26:06,106 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:06,106 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-10-11 17:26:16,117 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 72 2017-10-11 17:26:16,117 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:16,117 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 73 to run in 10 seconds 2017-10-11 17:26:20,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:20,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:20,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:20,995 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:20,995 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:26:20,995 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:26:20,995 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:26:20,995 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:26:20,995 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:26:21,007 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:26:21,007 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:26:21,008 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:26:21,008 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:26:21,008 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:26:21,008 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:26:21,008 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:26:21,008 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:26:26,124 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 73 2017-10-11 17:26:26,124 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:26,124 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-10-11 17:26:35,821 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 67 2017-10-11 17:26:35,822 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:26:35,822 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 75 to run in 60 seconds 2017-10-11 17:26:35,822 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 9 2017-10-11 17:26:35,823 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:26:35,823 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-10-11 17:26:35,871 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 68 2017-10-11 17:26:35,872 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 76 to run in 60 seconds 2017-10-11 17:26:36,002 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:36,003 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:36,003 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:36,003 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:36,003 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:36,003 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:36,003 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43057408 2017-10-11 17:26:36,004 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138848 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:36,004 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:26:36,004 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:36,004 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:36,004 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43064688 2017-10-11 17:26:36,004 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:36,005 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:26:36,005 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:36,005 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:36,005 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43252240 2017-10-11 17:26:36,005 | TRACE | remote.py ( 86) | connect | connecting socket 82 43244656 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:36,005 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:26:36,005 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:36,006 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:36,006 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43262688 2017-10-11 17:26:36,006 | TRACE | remote.py ( 86) | connect | connecting socket 90 43246304 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:36,006 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:26:36,006 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:26:36,017 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:26:36,018 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:26:36,018 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:26:36,018 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:26:36,019 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:26:36,019 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:26:36,019 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:26:36,019 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:26:36,126 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 74 2017-10-11 17:26:36,127 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:36,127 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 77 to run in 10 seconds 2017-10-11 17:26:46,132 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 77 2017-10-11 17:26:46,133 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:46,133 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-10-11 17:26:51,006 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:26:51,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:26:51,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:26:51,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:26:51,007 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:26:51,008 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:26:51,008 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:26:51,008 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:26:51,008 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:26:51,019 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:26:51,019 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:26:51,020 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:26:51,020 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:26:51,020 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:26:51,020 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:26:51,020 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:26:51,020 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:26:56,143 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 78 2017-10-11 17:26:56,143 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:26:56,143 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-10-11 17:27:06,016 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:06,016 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:06,016 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:06,017 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:06,017 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:06,017 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:06,017 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138848 2017-10-11 17:27:06,018 | TRACE | remote.py ( 86) | connect | connecting socket 84 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:06,019 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:27:06,019 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:06,019 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:06,019 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:27:06,020 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138848 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:06,020 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:27:06,020 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:06,020 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:06,021 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43244656 2017-10-11 17:27:06,021 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:06,021 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:27:06,022 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:06,022 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:06,022 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 90 43246304 2017-10-11 17:27:06,022 | TRACE | remote.py ( 86) | connect | connecting socket 83 43262688 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:06,023 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:27:06,023 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:27:06,034 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:27:06,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:27:06,035 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:27:06,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:27:06,035 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:27:06,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:27:06,036 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:27:06,036 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:27:06,150 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 79 2017-10-11 17:27:06,150 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:06,150 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 80 to run in 10 seconds 2017-10-11 17:27:16,155 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 80 2017-10-11 17:27:16,156 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:16,156 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-10-11 17:27:21,021 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:21,022 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:21,022 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:21,022 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:21,022 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:27:21,022 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:27:21,022 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:27:21,022 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:27:21,022 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:27:21,034 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:27:21,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:27:21,035 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:27:21,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:27:21,035 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:27:21,035 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:27:21,035 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:27:21,036 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:27:26,164 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 81 2017-10-11 17:27:26,164 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:26,164 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-10-11 17:27:35,830 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 75 2017-10-11 17:27:35,831 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:27:35,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 83 to run in 60 seconds 2017-10-11 17:27:35,831 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 10 2017-10-11 17:27:35,831 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:27:35,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-10-11 17:27:35,878 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 76 2017-10-11 17:27:35,878 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 84 to run in 60 seconds 2017-10-11 17:27:36,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:36,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:36,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:36,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:36,030 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:36,031 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:36,031 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43064688 2017-10-11 17:27:36,031 | TRACE | remote.py ( 86) | connect | connecting socket 82 43244656 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:36,031 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:27:36,031 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:36,031 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:36,031 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138848 2017-10-11 17:27:36,032 | TRACE | remote.py ( 86) | connect | connecting socket 84 43247424 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:36,032 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:27:36,032 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:36,032 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:36,032 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:27:36,032 | TRACE | remote.py ( 86) | connect | connecting socket 78 43064688 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:36,032 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:27:36,033 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:36,033 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:36,033 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43262688 2017-10-11 17:27:36,033 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138848 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:36,033 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:27:36,033 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:27:36,047 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:27:36,047 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:27:36,047 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:27:36,048 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:27:36,048 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:27:36,048 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:27:36,048 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:27:36,049 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:27:36,164 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 82 2017-10-11 17:27:36,165 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:36,165 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 85 to run in 10 seconds 2017-10-11 17:27:46,170 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 85 2017-10-11 17:27:46,171 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:46,171 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-10-11 17:27:51,039 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:27:51,039 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:27:51,039 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:27:51,039 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:27:51,040 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:27:51,040 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:27:51,040 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:27:51,040 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:27:51,040 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:27:51,052 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:27:51,052 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:27:51,052 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:27:51,052 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:27:51,052 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:27:51,052 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:27:51,052 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:27:51,053 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:27:56,180 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 86 2017-10-11 17:27:56,181 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:27:56,181 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 87 to run in 10 seconds 2017-10-11 17:28:06,049 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:06,050 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:06,050 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:06,050 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:06,051 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:06,051 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:06,051 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43244656 2017-10-11 17:28:06,051 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:06,051 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:28:06,052 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:06,052 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:06,052 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43247424 2017-10-11 17:28:06,052 | TRACE | remote.py ( 86) | connect | connecting socket 82 43244656 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:06,052 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:28:06,053 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:06,053 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:06,053 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43064688 2017-10-11 17:28:06,053 | TRACE | remote.py ( 86) | connect | connecting socket 86 43262688 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:06,053 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:28:06,054 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:06,055 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:06,055 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138848 2017-10-11 17:28:06,055 | TRACE | remote.py ( 86) | connect | connecting socket 88 43253360 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:06,055 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:28:06,058 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:28:06,070 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:28:06,070 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:28:06,071 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:28:06,071 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:28:06,071 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:28:06,071 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:28:06,071 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:28:06,071 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:28:06,186 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 87 2017-10-11 17:28:06,186 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:06,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-10-11 17:28:16,197 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 88 2017-10-11 17:28:16,197 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:16,198 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-10-11 17:28:21,050 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:21,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:21,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:21,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:21,051 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:28:21,051 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:28:21,051 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:28:21,051 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:28:21,051 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:28:21,064 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:28:21,064 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:28:21,064 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:28:21,064 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:28:21,064 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:28:21,064 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:28:21,065 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:28:21,065 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:28:26,208 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 89 2017-10-11 17:28:26,209 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:26,209 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-10-11 17:28:35,843 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 83 2017-10-11 17:28:35,843 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:28:35,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 91 to run in 60 seconds 2017-10-11 17:28:35,844 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 11 2017-10-11 17:28:35,844 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:28:35,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-10-11 17:28:35,879 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 84 2017-10-11 17:28:35,880 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 92 to run in 60 seconds 2017-10-11 17:28:36,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:36,052 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:36,053 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:36,053 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:36,053 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:36,054 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:36,054 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:28:36,055 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:36,055 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:28:36,055 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:36,056 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:36,056 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43244656 2017-10-11 17:28:36,056 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:36,057 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:28:36,057 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:36,057 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:36,057 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43262688 2017-10-11 17:28:36,058 | TRACE | remote.py ( 86) | connect | connecting socket 82 43138848 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:36,058 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:28:36,058 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:36,059 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:36,059 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43253360 2017-10-11 17:28:36,059 | TRACE | remote.py ( 86) | connect | connecting socket 87 43244656 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:36,059 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:28:36,060 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:28:36,071 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:28:36,071 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:28:36,072 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:28:36,072 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:28:36,072 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:28:36,073 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:28:36,073 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:28:36,073 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:28:36,215 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 90 2017-10-11 17:28:36,215 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:36,215 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 93 to run in 10 seconds 2017-10-11 17:28:46,226 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 93 2017-10-11 17:28:46,228 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:46,228 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 94 to run in 10 seconds 2017-10-11 17:28:51,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:28:51,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:28:51,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:28:51,062 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:28:51,062 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:28:51,063 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:28:51,063 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:28:51,063 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:28:51,063 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:28:51,074 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:28:51,075 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:28:51,075 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:28:51,075 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:28:51,075 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:28:51,075 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:28:51,075 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:28:51,075 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:28:56,245 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 94 2017-10-11 17:28:56,245 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:28:56,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-10-11 17:29:06,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:06,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:06,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:06,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:06,064 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:06,064 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:06,064 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064688 2017-10-11 17:29:06,065 | TRACE | remote.py ( 86) | connect | connecting socket 86 43253360 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:06,065 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:29:06,065 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:06,065 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:06,065 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:29:06,066 | TRACE | remote.py ( 86) | connect | connecting socket 88 43262688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:06,066 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:29:06,066 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:06,066 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:06,066 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43138848 2017-10-11 17:29:06,066 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064688 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:06,066 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:29:06,067 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:06,067 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:06,067 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43244656 2017-10-11 17:29:06,067 | TRACE | remote.py ( 86) | connect | connecting socket 83 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:06,067 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:29:06,067 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:29:06,079 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:29:06,080 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:29:06,080 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:29:06,080 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:29:06,080 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:29:06,080 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:29:06,080 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:29:06,080 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:29:06,249 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 95 2017-10-11 17:29:06,249 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:06,249 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-10-11 17:29:16,261 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 96 2017-10-11 17:29:16,261 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:16,261 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-10-11 17:29:21,067 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:21,067 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:21,068 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:21,068 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:21,068 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:29:21,069 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:29:21,069 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:29:21,069 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:29:21,070 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:29:21,081 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:29:21,081 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:29:21,081 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:29:21,082 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:29:21,082 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:29:21,082 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:29:21,083 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:29:21,083 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:29:26,273 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 97 2017-10-11 17:29:26,273 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:26,273 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-10-11 17:29:35,846 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 91 2017-10-11 17:29:35,846 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:29:35,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 99 to run in 60 seconds 2017-10-11 17:29:35,846 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 12 2017-10-11 17:29:35,846 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:29:35,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-10-11 17:29:35,881 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 92 2017-10-11 17:29:35,882 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 100 to run in 60 seconds 2017-10-11 17:29:36,077 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:36,077 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:36,077 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:36,077 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:36,077 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:36,078 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:36,078 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43253360 2017-10-11 17:29:36,078 | TRACE | remote.py ( 86) | connect | connecting socket 78 43139280 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:36,078 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:29:36,078 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:36,078 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:36,079 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43262688 2017-10-11 17:29:36,079 | TRACE | remote.py ( 86) | connect | connecting socket 85 43253360 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:36,080 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:29:36,081 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:36,081 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:36,081 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064688 2017-10-11 17:29:36,082 | TRACE | remote.py ( 86) | connect | connecting socket 87 43262688 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:36,082 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:29:36,082 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:36,083 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:36,083 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43057408 2017-10-11 17:29:36,083 | TRACE | remote.py ( 86) | connect | connecting socket 88 43244656 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:36,083 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:29:36,083 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:29:36,098 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:29:36,098 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:29:36,098 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:29:36,099 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:29:36,099 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:29:36,099 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:29:36,099 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:29:36,099 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:29:36,281 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 98 2017-10-11 17:29:36,282 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:36,282 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 101 to run in 10 seconds 2017-10-11 17:29:46,292 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 101 2017-10-11 17:29:46,292 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:46,293 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 102 to run in 10 seconds 2017-10-11 17:29:51,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:29:51,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:29:51,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:29:51,088 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:29:51,088 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:29:51,089 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:29:51,089 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:29:51,089 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:29:51,090 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:29:51,101 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:29:51,102 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:29:51,102 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:29:51,102 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:29:51,103 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:29:51,103 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:29:51,103 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:29:51,103 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:29:56,301 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 102 2017-10-11 17:29:56,301 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:29:56,302 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 103 to run in 10 seconds 2017-10-11 17:30:06,094 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:06,095 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:06,095 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:06,095 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:06,095 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:06,096 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:06,096 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43139280 2017-10-11 17:30:06,097 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:06,098 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:30:06,099 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:06,099 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:06,099 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43253360 2017-10-11 17:30:06,099 | TRACE | remote.py ( 86) | connect | connecting socket 81 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:06,099 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:30:06,099 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:06,100 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:06,100 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43262688 2017-10-11 17:30:06,100 | TRACE | remote.py ( 86) | connect | connecting socket 82 43139280 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:06,100 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:30:06,100 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:06,103 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:06,103 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43244656 2017-10-11 17:30:06,103 | TRACE | remote.py ( 86) | connect | connecting socket 83 43253360 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:06,105 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:30:06,105 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:30:06,118 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:30:06,118 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:30:06,118 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:30:06,118 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:30:06,119 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:30:06,119 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:30:06,119 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:30:06,119 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:30:06,311 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 103 2017-10-11 17:30:06,312 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:06,312 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 104 to run in 10 seconds 2017-10-11 17:30:16,318 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 104 2017-10-11 17:30:16,318 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:16,318 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 105 to run in 10 seconds 2017-10-11 17:30:21,100 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:21,100 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:21,100 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:21,100 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:21,100 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:30:21,100 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:30:21,101 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:30:21,101 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:30:21,101 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:30:21,112 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:30:21,113 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:30:21,113 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:30:21,113 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:30:21,113 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:30:21,113 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:30:21,113 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:30:21,114 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:30:26,322 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 105 2017-10-11 17:30:26,323 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:26,324 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 106 to run in 10 seconds 2017-10-11 17:30:35,856 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 99 2017-10-11 17:30:35,856 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:30:35,857 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 107 to run in 60 seconds 2017-10-11 17:30:35,857 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 13 2017-10-11 17:30:35,857 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:30:35,858 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-10-11 17:30:35,894 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 100 2017-10-11 17:30:35,895 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 108 to run in 60 seconds 2017-10-11 17:30:36,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:36,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:36,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:36,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:36,106 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:36,106 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:36,106 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064688 2017-10-11 17:30:36,106 | TRACE | remote.py ( 86) | connect | connecting socket 86 43262928 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:36,106 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:30:36,106 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:36,107 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:36,107 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43057408 2017-10-11 17:30:36,107 | TRACE | remote.py ( 86) | connect | connecting socket 87 43249664 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:36,107 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:30:36,107 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:36,107 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:36,108 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43139280 2017-10-11 17:30:36,108 | TRACE | remote.py ( 86) | connect | connecting socket 90 43244656 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:36,108 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:30:36,108 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:36,108 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:36,108 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43253360 2017-10-11 17:30:36,108 | TRACE | remote.py ( 86) | connect | connecting socket 93 43145040 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:36,109 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:30:36,109 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:30:36,121 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:30:36,121 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:30:36,121 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:30:36,121 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:30:36,121 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:30:36,121 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:30:36,121 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:30:36,123 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:30:36,327 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 106 2017-10-11 17:30:36,327 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:36,327 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 109 to run in 10 seconds 2017-10-11 17:30:46,330 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 109 2017-10-11 17:30:46,331 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:46,331 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 110 to run in 10 seconds 2017-10-11 17:30:51,110 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:30:51,110 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:30:51,111 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:30:51,111 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:30:51,111 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:30:51,111 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:30:51,111 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:30:51,111 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:30:51,111 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:30:51,123 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:30:51,123 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:30:51,123 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:30:51,123 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:30:51,123 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:30:51,123 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:30:51,124 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:30:51,124 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:30:56,340 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 110 2017-10-11 17:30:56,340 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:30:56,340 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 111 to run in 10 seconds 2017-10-11 17:31:06,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:06,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:06,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:06,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:06,113 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:06,114 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:06,114 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43262928 2017-10-11 17:31:06,114 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:06,114 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:31:06,114 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:06,114 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:06,115 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43249664 2017-10-11 17:31:06,115 | TRACE | remote.py ( 86) | connect | connecting socket 82 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:06,115 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:31:06,115 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:06,115 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:06,115 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 90 43244656 2017-10-11 17:31:06,116 | TRACE | remote.py ( 86) | connect | connecting socket 84 43139280 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:06,116 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:31:06,116 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:06,116 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:06,116 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 93 43145040 2017-10-11 17:31:06,116 | TRACE | remote.py ( 86) | connect | connecting socket 85 43248544 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:06,116 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:31:06,116 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:31:06,128 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:31:06,129 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:31:06,129 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:31:06,129 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:31:06,129 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:31:06,129 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:31:06,129 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:31:06,129 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:31:06,348 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 111 2017-10-11 17:31:06,349 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:06,349 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 112 to run in 10 seconds 2017-10-11 17:31:16,357 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 112 2017-10-11 17:31:16,357 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:16,357 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 113 to run in 10 seconds 2017-10-11 17:31:21,120 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:21,121 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:21,121 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:21,121 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:21,121 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:31:21,122 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:31:21,122 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:31:21,123 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:31:21,123 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:31:21,134 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:31:21,135 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:31:21,135 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:31:21,135 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:31:21,136 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:31:21,136 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:31:21,136 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:31:21,136 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:31:26,365 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 113 2017-10-11 17:31:26,365 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:26,365 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 114 to run in 10 seconds 2017-10-11 17:31:35,858 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 107 2017-10-11 17:31:35,859 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:31:35,859 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 115 to run in 60 seconds 2017-10-11 17:31:35,860 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 14 2017-10-11 17:31:35,860 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:31:35,860 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 15 to run in 60 seconds 2017-10-11 17:31:35,896 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 108 2017-10-11 17:31:35,898 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 116 to run in 60 seconds 2017-10-11 17:31:36,128 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:36,129 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:36,129 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:36,129 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:36,130 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:36,130 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:36,130 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064688 2017-10-11 17:31:36,131 | TRACE | remote.py ( 86) | connect | connecting socket 78 43145040 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:36,134 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:31:36,135 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:36,135 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:36,136 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43057408 2017-10-11 17:31:36,137 | TRACE | remote.py ( 86) | connect | connecting socket 81 43064688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:36,138 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:31:36,138 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:36,138 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:36,138 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43139280 2017-10-11 17:31:36,142 | TRACE | remote.py ( 86) | connect | connecting socket 83 43139280 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:36,142 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:31:36,142 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:36,142 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:36,142 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43248544 2017-10-11 17:31:36,143 | TRACE | remote.py ( 86) | connect | connecting socket 84 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:36,143 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:31:36,143 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:31:36,155 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:31:36,155 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:31:36,155 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:31:36,155 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:31:36,155 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:31:36,155 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:31:36,155 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:31:36,156 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:31:36,370 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 114 2017-10-11 17:31:36,371 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:36,371 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 117 to run in 10 seconds 2017-10-11 17:31:46,379 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 117 2017-10-11 17:31:46,379 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:46,379 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 118 to run in 10 seconds 2017-10-11 17:31:51,131 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:31:51,132 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:31:51,132 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:31:51,132 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:31:51,132 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:31:51,132 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:31:51,132 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:31:51,133 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:31:51,133 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:31:51,145 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:31:51,145 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:31:51,145 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:31:51,145 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:31:51,145 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:31:51,145 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:31:51,145 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:31:51,145 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:31:56,389 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 118 2017-10-11 17:31:56,389 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:31:56,389 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 119 to run in 10 seconds 2017-10-11 17:32:06,142 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:06,142 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:06,142 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:06,143 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:06,143 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:06,143 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:06,143 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43145040 2017-10-11 17:32:06,146 | TRACE | remote.py ( 86) | connect | connecting socket 87 43254480 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:06,147 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:32:06,147 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:06,147 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:06,147 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43064688 2017-10-11 17:32:06,147 | TRACE | remote.py ( 86) | connect | connecting socket 78 43145040 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:06,148 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:32:06,148 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:06,148 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:06,148 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43139280 2017-10-11 17:32:06,148 | TRACE | remote.py ( 86) | connect | connecting socket 79 43246304 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:06,148 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:32:06,149 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:06,149 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:06,149 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43057408 2017-10-11 17:32:06,149 | TRACE | remote.py ( 86) | connect | connecting socket 89 43249072 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:06,149 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:32:06,149 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:32:06,163 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:32:06,163 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:32:06,164 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:32:06,164 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:32:06,164 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:32:06,164 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:32:06,164 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:32:06,164 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:32:06,398 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 119 2017-10-11 17:32:06,399 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:06,399 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 120 to run in 10 seconds 2017-10-11 17:32:16,407 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 120 2017-10-11 17:32:16,407 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:16,407 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 121 to run in 10 seconds 2017-10-11 17:32:21,153 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:21,154 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:21,154 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:21,154 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:21,155 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:32:21,155 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:32:21,155 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:32:21,156 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:32:21,156 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:32:21,167 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:32:21,168 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:32:21,168 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:32:21,168 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:32:21,168 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:32:21,169 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:32:21,169 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:32:21,169 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:32:26,413 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 121 2017-10-11 17:32:26,413 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:26,414 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 122 to run in 10 seconds 2017-10-11 17:32:35,866 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 115 2017-10-11 17:32:35,866 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:32:35,866 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 123 to run in 60 seconds 2017-10-11 17:32:35,866 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 15 2017-10-11 17:32:35,866 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:32:35,866 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 16 to run in 60 seconds 2017-10-11 17:32:35,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 116 2017-10-11 17:32:35,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 124 to run in 60 seconds 2017-10-11 17:32:36,156 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:36,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:36,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:36,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:36,157 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:36,158 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:36,158 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43254480 2017-10-11 17:32:36,160 | TRACE | remote.py ( 86) | connect | connecting socket 81 43064688 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:36,160 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:32:36,160 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:36,161 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:36,161 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43145040 2017-10-11 17:32:36,161 | TRACE | remote.py ( 86) | connect | connecting socket 84 43057408 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:36,162 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:32:36,162 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:36,162 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:36,163 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43246304 2017-10-11 17:32:36,163 | TRACE | remote.py ( 86) | connect | connecting socket 85 43252240 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:36,163 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:32:36,164 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:36,164 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:36,164 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43249072 2017-10-11 17:32:36,165 | TRACE | remote.py ( 86) | connect | connecting socket 88 43255008 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:36,165 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:32:36,165 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:32:36,177 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:32:36,177 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:32:36,178 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:32:36,178 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:32:36,178 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:32:36,179 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:32:36,179 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:32:36,179 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:32:36,425 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 122 2017-10-11 17:32:36,426 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:36,426 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 125 to run in 10 seconds 2017-10-11 17:32:46,438 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 125 2017-10-11 17:32:46,438 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:46,438 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 126 to run in 10 seconds 2017-10-11 17:32:51,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:32:51,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:32:51,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:32:51,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:32:51,167 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:32:51,168 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:32:51,168 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:32:51,168 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:32:51,168 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:32:51,180 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:32:51,180 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:32:51,180 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:32:51,180 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:32:51,180 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:32:51,181 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:32:51,181 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:32:51,181 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:32:56,449 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 126 2017-10-11 17:32:56,449 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:32:56,449 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 127 to run in 10 seconds 2017-10-11 17:33:06,169 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:06,170 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:06,170 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:06,171 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:06,171 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:06,171 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:06,171 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43064688 2017-10-11 17:33:06,172 | TRACE | remote.py ( 86) | connect | connecting socket 79 43145040 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:06,172 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:33:06,172 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:06,172 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:06,172 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43057408 2017-10-11 17:33:06,173 | TRACE | remote.py ( 86) | connect | connecting socket 81 43064688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:06,173 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:33:06,173 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:06,173 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:06,173 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43252240 2017-10-11 17:33:06,174 | TRACE | remote.py ( 86) | connect | connecting socket 84 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:06,174 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:33:06,174 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:06,174 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:06,174 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43255008 2017-10-11 17:33:06,175 | TRACE | remote.py ( 86) | connect | connecting socket 86 43246304 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:06,175 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:33:06,175 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:33:06,188 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:33:06,188 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:33:06,188 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:33:06,188 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:33:06,188 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:33:06,188 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:33:06,189 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:33:06,189 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:33:06,453 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 127 2017-10-11 17:33:06,453 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:06,453 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 128 to run in 10 seconds 2017-10-11 17:33:16,465 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 128 2017-10-11 17:33:16,465 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:16,466 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 129 to run in 10 seconds 2017-10-11 17:33:21,172 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:21,172 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:21,172 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:21,173 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:21,173 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:33:21,173 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:33:21,173 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:33:21,173 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:33:21,173 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:33:21,185 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:33:21,186 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:33:21,187 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:33:21,187 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:33:21,187 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:33:21,188 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:33:21,188 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:33:21,188 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:33:26,473 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 129 2017-10-11 17:33:26,473 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:26,473 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 130 to run in 10 seconds 2017-10-11 17:33:35,872 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 123 2017-10-11 17:33:35,872 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:33:35,872 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 131 to run in 60 seconds 2017-10-11 17:33:35,873 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 16 2017-10-11 17:33:35,873 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:33:35,873 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 17 to run in 60 seconds 2017-10-11 17:33:35,910 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 124 2017-10-11 17:33:35,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 132 to run in 60 seconds 2017-10-11 17:33:36,175 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:36,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:36,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:36,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:36,177 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:36,177 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:36,178 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43145040 2017-10-11 17:33:36,179 | TRACE | remote.py ( 86) | connect | connecting socket 78 43249072 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:36,179 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:33:36,179 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:36,179 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:36,179 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43064688 2017-10-11 17:33:36,179 | TRACE | remote.py ( 86) | connect | connecting socket 85 43250720 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:36,180 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:33:36,180 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:36,180 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:36,180 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43057408 2017-10-11 17:33:36,180 | TRACE | remote.py ( 86) | connect | connecting socket 79 43138608 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:36,181 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:33:36,181 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:36,181 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:36,181 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43246304 2017-10-11 17:33:36,181 | TRACE | remote.py ( 86) | connect | connecting socket 82 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:36,183 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:33:36,183 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:33:36,194 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:33:36,195 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:33:36,195 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:33:36,195 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:33:36,195 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:33:36,196 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:33:36,196 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:33:36,196 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:33:36,477 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 130 2017-10-11 17:33:36,477 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:36,477 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 133 to run in 10 seconds 2017-10-11 17:33:46,480 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 133 2017-10-11 17:33:46,481 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:46,481 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 134 to run in 10 seconds 2017-10-11 17:33:51,176 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:33:51,177 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:33:51,177 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:33:51,177 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:33:51,177 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:33:51,177 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:33:51,178 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:33:51,178 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:33:51,178 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:33:51,190 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:33:51,190 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:33:51,190 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:33:51,190 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:33:51,190 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:33:51,193 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:33:51,193 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:33:51,193 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:33:56,491 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 134 2017-10-11 17:33:56,492 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:33:56,492 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 135 to run in 10 seconds 2017-10-11 17:34:06,187 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:06,187 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:06,187 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:06,187 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:06,187 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:06,187 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:06,188 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43249072 2017-10-11 17:34:06,188 | TRACE | remote.py ( 86) | connect | connecting socket 84 43246304 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:06,188 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:34:06,188 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:06,189 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:06,189 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43250720 2017-10-11 17:34:06,189 | TRACE | remote.py ( 86) | connect | connecting socket 78 43249072 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:06,189 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:34:06,189 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:06,189 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:06,190 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43138608 2017-10-11 17:34:06,190 | TRACE | remote.py ( 86) | connect | connecting socket 85 43264048 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:06,191 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:34:06,191 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:06,192 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:06,192 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43057408 2017-10-11 17:34:06,192 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064896 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:06,192 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:34:06,192 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:34:06,204 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:34:06,204 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:34:06,204 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:34:06,204 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:34:06,204 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:34:06,204 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:34:06,204 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:34:06,205 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:34:06,503 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 135 2017-10-11 17:34:06,504 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:06,504 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 136 to run in 10 seconds 2017-10-11 17:34:16,512 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 136 2017-10-11 17:34:16,512 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:16,512 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 137 to run in 10 seconds 2017-10-11 17:34:21,192 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:21,192 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:21,192 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:21,193 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:21,193 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:34:21,193 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:34:21,193 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:34:21,193 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:34:21,193 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:34:21,205 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:34:21,206 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:34:21,206 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:34:21,206 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:34:21,206 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:34:21,206 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:34:21,207 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:34:21,207 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:34:26,520 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 137 2017-10-11 17:34:26,521 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:26,521 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 138 to run in 10 seconds 2017-10-11 17:34:35,879 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 131 2017-10-11 17:34:35,879 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:34:35,879 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 139 to run in 60 seconds 2017-10-11 17:34:35,879 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 17 2017-10-11 17:34:35,880 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:34:35,880 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 18 to run in 60 seconds 2017-10-11 17:34:35,914 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 132 2017-10-11 17:34:35,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 140 to run in 60 seconds 2017-10-11 17:34:36,195 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:36,195 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:36,195 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:36,195 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:36,195 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:36,195 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:36,196 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43246304 2017-10-11 17:34:36,196 | TRACE | remote.py ( 86) | connect | connecting socket 82 43057408 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:36,196 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:34:36,196 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:36,196 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:36,196 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43249072 2017-10-11 17:34:36,197 | TRACE | remote.py ( 86) | connect | connecting socket 84 43250720 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:36,197 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:34:36,197 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:36,197 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:36,197 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43264048 2017-10-11 17:34:36,197 | TRACE | remote.py ( 86) | connect | connecting socket 78 43253488 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:36,198 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:34:36,198 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:36,198 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:36,198 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064896 2017-10-11 17:34:36,198 | TRACE | remote.py ( 86) | connect | connecting socket 87 43255136 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:36,198 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:34:36,199 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:34:36,210 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:34:36,210 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:34:36,210 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:34:36,210 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:34:36,210 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:34:36,210 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:34:36,211 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:34:36,211 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:34:36,527 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 138 2017-10-11 17:34:36,527 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:36,527 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 141 to run in 10 seconds 2017-10-11 17:34:46,530 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 141 2017-10-11 17:34:46,530 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:46,530 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 142 to run in 10 seconds 2017-10-11 17:34:51,201 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:34:51,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:34:51,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:34:51,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:34:51,202 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:34:51,202 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:34:51,203 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:34:51,203 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:34:51,203 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:34:51,215 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:34:51,215 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:34:51,215 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:34:51,216 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:34:51,216 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:34:51,216 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:34:51,216 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:34:51,217 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:34:56,533 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 142 2017-10-11 17:34:56,533 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:34:56,533 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 143 to run in 10 seconds 2017-10-11 17:35:06,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:06,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:06,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:06,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:06,207 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:06,207 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:06,207 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43057408 2017-10-11 17:35:06,207 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064896 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:06,207 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:35:06,207 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:06,208 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:06,208 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43250720 2017-10-11 17:35:06,208 | TRACE | remote.py ( 86) | connect | connecting socket 82 43263840 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:06,208 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:35:06,208 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:06,208 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:06,208 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43253488 2017-10-11 17:35:06,208 | TRACE | remote.py ( 86) | connect | connecting socket 88 43057408 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:06,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:35:06,209 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:06,209 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:06,209 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43255136 2017-10-11 17:35:06,209 | TRACE | remote.py ( 86) | connect | connecting socket 89 43243536 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:06,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:35:06,209 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:35:06,221 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:35:06,221 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:35:06,221 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:35:06,222 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:35:06,222 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:35:06,222 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:35:06,224 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:35:06,224 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:35:06,540 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 143 2017-10-11 17:35:06,540 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:06,540 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 144 to run in 10 seconds 2017-10-11 17:35:16,544 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 144 2017-10-11 17:35:16,544 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:16,545 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 145 to run in 10 seconds 2017-10-11 17:35:21,207 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:21,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:21,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:21,208 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:21,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:35:21,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:35:21,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:35:21,209 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:35:21,210 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:35:21,221 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:35:21,221 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:35:21,221 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:35:21,221 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:35:21,222 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:35:21,222 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:35:21,222 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:35:21,222 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:35:26,552 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 145 2017-10-11 17:35:26,553 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:26,553 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 146 to run in 10 seconds 2017-10-11 17:35:35,888 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 139 2017-10-11 17:35:35,889 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:35:35,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 147 to run in 60 seconds 2017-10-11 17:35:35,889 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 18 2017-10-11 17:35:35,889 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:35:35,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 19 to run in 60 seconds 2017-10-11 17:35:35,925 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 140 2017-10-11 17:35:35,925 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 148 to run in 60 seconds 2017-10-11 17:35:36,214 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:36,215 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:36,215 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:36,215 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:36,216 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:36,216 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:36,216 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064896 2017-10-11 17:35:36,217 | TRACE | remote.py ( 86) | connect | connecting socket 78 43245184 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:36,217 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:35:36,217 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:36,217 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:36,218 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43263840 2017-10-11 17:35:36,218 | TRACE | remote.py ( 86) | connect | connecting socket 79 43064688 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:36,218 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:35:36,219 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:36,219 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:36,219 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43057408 2017-10-11 17:35:36,220 | TRACE | remote.py ( 86) | connect | connecting socket 83 43246832 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:36,220 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:35:36,220 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:36,220 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:36,221 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43243536 2017-10-11 17:35:36,221 | TRACE | remote.py ( 86) | connect | connecting socket 86 43249600 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:36,221 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:35:36,222 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:35:36,233 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:35:36,234 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:35:36,234 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:35:36,234 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:35:36,235 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:35:36,235 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:35:36,235 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:35:36,235 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:35:36,557 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 146 2017-10-11 17:35:36,558 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:36,558 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 149 to run in 10 seconds 2017-10-11 17:35:46,562 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 149 2017-10-11 17:35:46,562 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:46,562 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 150 to run in 10 seconds 2017-10-11 17:35:51,218 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:35:51,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:35:51,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:35:51,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:35:51,219 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:35:51,219 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:35:51,219 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:35:51,219 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:35:51,220 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:35:51,231 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:35:51,232 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:35:51,232 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:35:51,232 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:35:51,232 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:35:51,232 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:35:51,232 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:35:51,232 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:35:56,571 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 150 2017-10-11 17:35:56,571 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:35:56,571 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 151 to run in 10 seconds 2017-10-11 17:36:06,223 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:06,223 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:06,223 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:06,223 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:06,223 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:06,223 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:06,224 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43245184 2017-10-11 17:36:06,224 | TRACE | remote.py ( 86) | connect | connecting socket 82 43243536 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:06,225 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:36:06,225 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:06,225 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:06,226 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43064688 2017-10-11 17:36:06,226 | TRACE | remote.py ( 86) | connect | connecting socket 78 43245184 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:06,226 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:36:06,227 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:06,227 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:06,227 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43246832 2017-10-11 17:36:06,227 | TRACE | remote.py ( 86) | connect | connecting socket 88 43263840 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:06,227 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:36:06,227 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:06,228 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:06,228 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43249600 2017-10-11 17:36:06,228 | TRACE | remote.py ( 86) | connect | connecting socket 89 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:06,228 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:36:06,228 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:36:06,241 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:36:06,243 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:36:06,243 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:36:06,244 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:36:06,244 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:36:06,244 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:36:06,244 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:36:06,244 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:36:06,579 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 151 2017-10-11 17:36:06,579 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:06,579 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 152 to run in 10 seconds 2017-10-11 17:36:16,581 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 152 2017-10-11 17:36:16,582 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:16,582 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 153 to run in 10 seconds 2017-10-11 17:36:21,228 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:21,228 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:21,229 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:21,229 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:21,229 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:36:21,229 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:36:21,229 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:36:21,229 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:36:21,229 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:36:21,241 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:36:21,242 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:36:21,242 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:36:21,242 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:36:21,242 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:36:21,242 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:36:21,243 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:36:21,243 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:36:26,587 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 153 2017-10-11 17:36:26,588 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:26,588 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 154 to run in 10 seconds 2017-10-11 17:36:35,896 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 147 2017-10-11 17:36:35,897 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:36:35,897 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 155 to run in 60 seconds 2017-10-11 17:36:35,898 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 19 2017-10-11 17:36:35,898 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:36:35,898 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 20 to run in 60 seconds 2017-10-11 17:36:35,932 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 148 2017-10-11 17:36:35,933 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 156 to run in 60 seconds 2017-10-11 17:36:36,239 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:36,239 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:36,239 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:36,239 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:36,239 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:36,240 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:36,240 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43243536 2017-10-11 17:36:36,240 | TRACE | remote.py ( 86) | connect | connecting socket 79 43243536 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:36,240 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:36:36,240 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:36,241 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:36,241 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43245184 2017-10-11 17:36:36,241 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138608 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:36,242 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:36:36,242 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:36,242 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:36,242 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43263840 2017-10-11 17:36:36,242 | TRACE | remote.py ( 86) | connect | connecting socket 83 43263840 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:36,242 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:36:36,243 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:36,243 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:36,243 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43057408 2017-10-11 17:36:36,244 | TRACE | remote.py ( 86) | connect | connecting socket 85 43057408 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:36,244 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:36:36,245 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:36:36,256 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:36:36,257 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:36:36,257 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:36:36,258 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:36:36,258 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:36:36,258 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:36:36,259 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:36:36,259 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:36:36,590 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 154 2017-10-11 17:36:36,591 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:36,591 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 157 to run in 10 seconds 2017-10-11 17:36:46,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 157 2017-10-11 17:36:46,596 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:46,596 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 158 to run in 10 seconds 2017-10-11 17:36:51,247 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:36:51,247 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:36:51,247 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:36:51,247 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:36:51,248 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:36:51,248 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:36:51,248 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:36:51,248 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:36:51,248 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:36:51,260 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:36:51,260 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:36:51,261 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:36:51,261 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:36:51,261 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:36:51,261 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:36:51,261 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:36:51,261 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:36:56,602 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 158 2017-10-11 17:36:56,602 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:36:56,603 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 159 to run in 10 seconds 2017-10-11 17:37:06,249 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:06,249 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:06,249 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:06,250 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:06,250 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:06,250 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:06,250 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43243536 2017-10-11 17:37:06,251 | TRACE | remote.py ( 86) | connect | connecting socket 79 43284016 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:06,251 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:37:06,251 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:06,251 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:06,251 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138608 2017-10-11 17:37:06,256 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138608 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:06,257 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:37:06,257 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:06,257 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:06,257 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43263840 2017-10-11 17:37:06,258 | TRACE | remote.py ( 86) | connect | connecting socket 87 43065120 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:06,258 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:37:06,258 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:06,258 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:06,258 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 85 43057408 2017-10-11 17:37:06,259 | TRACE | remote.py ( 86) | connect | connecting socket 88 43319616 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:06,259 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:37:06,259 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:37:06,271 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:37:06,272 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:37:06,277 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:37:06,278 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:37:06,278 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:37:06,278 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:37:06,279 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:37:06,279 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:37:06,605 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 159 2017-10-11 17:37:06,606 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:06,607 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 160 to run in 10 seconds 2017-10-11 17:37:16,607 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 160 2017-10-11 17:37:16,608 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:16,608 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 161 to run in 10 seconds 2017-10-11 17:37:21,254 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:21,254 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:21,254 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:21,254 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:21,255 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:37:21,255 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:37:21,255 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:37:21,255 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:37:21,255 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:37:21,267 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:37:21,268 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:37:21,268 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:37:21,268 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:37:21,270 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:37:21,270 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:37:21,270 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:37:21,270 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:37:26,616 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 161 2017-10-11 17:37:26,617 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:26,617 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 162 to run in 10 seconds 2017-10-11 17:37:35,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 155 2017-10-11 17:37:35,905 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:37:35,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 163 to run in 60 seconds 2017-10-11 17:37:35,905 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 20 2017-10-11 17:37:35,905 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:37:35,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 21 to run in 60 seconds 2017-10-11 17:37:35,943 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 156 2017-10-11 17:37:35,944 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 164 to run in 60 seconds 2017-10-11 17:37:36,260 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:36,261 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:36,261 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:36,261 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:36,262 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:36,262 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:36,262 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43284016 2017-10-11 17:37:36,263 | TRACE | remote.py ( 86) | connect | connecting socket 83 43263840 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:36,263 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:37:36,264 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:36,264 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:36,264 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138608 2017-10-11 17:37:36,264 | TRACE | remote.py ( 86) | connect | connecting socket 79 43284016 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:36,265 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:37:36,265 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:36,265 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:36,266 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43065120 2017-10-11 17:37:36,266 | TRACE | remote.py ( 86) | connect | connecting socket 81 43243536 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:36,267 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:37:36,268 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:36,268 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:36,271 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 88 43319616 2017-10-11 17:37:36,271 | TRACE | remote.py ( 86) | connect | connecting socket 78 43138608 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:36,272 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:37:36,272 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:37:36,285 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:37:36,286 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:37:36,286 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:37:36,287 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:37:36,287 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:37:36,287 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:37:36,287 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:37:36,288 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:37:36,622 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 162 2017-10-11 17:37:36,622 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:36,623 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 165 to run in 10 seconds 2017-10-11 17:37:46,625 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 165 2017-10-11 17:37:46,625 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:46,625 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 166 to run in 10 seconds 2017-10-11 17:37:51,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:37:51,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:37:51,271 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:37:51,272 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:37:51,272 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:37:51,272 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:37:51,272 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:37:51,272 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:37:51,272 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:37:51,284 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:37:51,284 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:37:51,284 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:37:51,284 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:37:51,287 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:37:51,288 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:37:51,288 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:37:51,288 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:37:56,630 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 166 2017-10-11 17:37:56,630 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:37:56,631 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 167 to run in 10 seconds 2017-10-11 17:38:06,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:06,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:06,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:06,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:06,282 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:06,282 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:06,282 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43263840 2017-10-11 17:38:06,282 | TRACE | remote.py ( 86) | connect | connecting socket 87 43065120 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:06,282 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:38:06,283 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:06,283 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:06,283 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 79 43284016 2017-10-11 17:38:06,283 | TRACE | remote.py ( 86) | connect | connecting socket 89 43246304 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:06,283 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:38:06,283 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:06,283 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:06,284 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43243536 2017-10-11 17:38:06,284 | TRACE | remote.py ( 86) | connect | connecting socket 91 43318496 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:06,284 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:38:06,284 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:06,284 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:06,284 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43138608 2017-10-11 17:38:06,286 | TRACE | remote.py ( 86) | connect | connecting socket 93 43249072 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:06,286 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:38:06,286 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:38:06,302 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:38:06,303 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:38:06,303 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:38:06,303 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:38:06,304 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:38:06,304 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:38:06,304 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:38:06,305 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:38:06,632 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 167 2017-10-11 17:38:06,632 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:06,632 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 168 to run in 10 seconds 2017-10-11 17:38:16,636 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 168 2017-10-11 17:38:16,636 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:16,636 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 169 to run in 10 seconds 2017-10-11 17:38:21,284 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:21,284 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:21,285 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:21,285 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:21,285 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:38:21,285 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:38:21,285 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:38:21,285 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:38:21,285 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:38:21,296 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:38:21,297 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:38:21,297 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:38:21,297 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:38:21,297 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:38:21,297 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:38:21,297 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:38:21,297 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:38:26,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 169 2017-10-11 17:38:26,639 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:26,639 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 170 to run in 10 seconds 2017-10-11 17:38:35,914 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 163 2017-10-11 17:38:35,914 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:38:35,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 171 to run in 60 seconds 2017-10-11 17:38:35,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 21 2017-10-11 17:38:35,915 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:38:35,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 22 to run in 60 seconds 2017-10-11 17:38:35,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 164 2017-10-11 17:38:35,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 172 to run in 60 seconds 2017-10-11 17:38:36,287 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:36,287 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:36,287 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:36,287 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:36,287 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:36,288 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:36,288 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43065120 2017-10-11 17:38:36,288 | TRACE | remote.py ( 86) | connect | connecting socket 78 43284016 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:36,288 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:38:36,289 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:36,289 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:36,289 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 89 43246304 2017-10-11 17:38:36,289 | TRACE | remote.py ( 86) | connect | connecting socket 81 43065120 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:36,289 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:38:36,290 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:36,290 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:36,291 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 91 43318496 2017-10-11 17:38:36,292 | TRACE | remote.py ( 86) | connect | connecting socket 83 43138608 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:36,292 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:38:36,292 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:36,292 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:36,292 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 93 43249072 2017-10-11 17:38:36,293 | TRACE | remote.py ( 86) | connect | connecting socket 84 43263840 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:36,293 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:38:36,293 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:38:36,307 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:38:36,308 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:38:36,308 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:38:36,308 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:38:36,308 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:38:36,308 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:38:36,308 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:38:36,308 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:38:36,649 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 170 2017-10-11 17:38:36,650 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:36,650 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 173 to run in 10 seconds 2017-10-11 17:38:46,655 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 173 2017-10-11 17:38:46,655 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:46,655 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 174 to run in 10 seconds 2017-10-11 17:38:51,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:38:51,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:38:51,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:38:51,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:38:51,293 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:38:51,294 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:38:51,294 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:38:51,294 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:38:51,294 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:38:51,305 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:38:51,306 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:38:51,306 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:38:51,306 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:38:51,306 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:38:51,306 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:38:51,307 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:38:51,307 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:38:56,662 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 174 2017-10-11 17:38:56,662 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:38:56,662 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 175 to run in 10 seconds 2017-10-11 17:39:06,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:06,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:06,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:06,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:06,296 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:06,297 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:06,297 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43284016 2017-10-11 17:39:06,297 | TRACE | remote.py ( 86) | connect | connecting socket 87 43318496 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:06,297 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:39:06,297 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:06,298 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:06,298 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43065120 2017-10-11 17:39:06,299 | TRACE | remote.py ( 86) | connect | connecting socket 78 43284016 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:06,299 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:39:06,300 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:06,300 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:06,300 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43138608 2017-10-11 17:39:06,300 | TRACE | remote.py ( 86) | connect | connecting socket 81 43065120 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:06,300 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:39:06,300 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:06,300 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:06,301 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43263840 2017-10-11 17:39:06,301 | TRACE | remote.py ( 86) | connect | connecting socket 82 43210704 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:06,301 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:39:06,301 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:39:06,312 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:39:06,313 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:39:06,313 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:39:06,313 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:39:06,313 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:39:06,313 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:39:06,313 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:39:06,313 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:39:06,671 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 175 2017-10-11 17:39:06,671 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:06,671 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 176 to run in 10 seconds 2017-10-11 17:39:16,676 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 176 2017-10-11 17:39:16,676 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:16,676 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 177 to run in 10 seconds 2017-10-11 17:39:21,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:21,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:21,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:21,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:21,307 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:39:21,307 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:39:21,307 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:39:21,307 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:39:21,308 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:39:21,319 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:39:21,319 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:39:21,319 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:39:21,319 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:39:21,319 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:39:21,320 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:39:21,320 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:39:21,320 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:39:26,686 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 177 2017-10-11 17:39:26,686 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:26,686 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 178 to run in 10 seconds 2017-10-11 17:39:35,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 171 2017-10-11 17:39:35,915 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:39:35,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 179 to run in 60 seconds 2017-10-11 17:39:35,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 22 2017-10-11 17:39:35,915 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:39:35,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 23 to run in 60 seconds 2017-10-11 17:39:35,951 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 172 2017-10-11 17:39:35,952 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 180 to run in 60 seconds 2017-10-11 17:39:36,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:36,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:36,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:36,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:36,315 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:36,316 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:36,316 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 87 43318496 2017-10-11 17:39:36,316 | TRACE | remote.py ( 86) | connect | connecting socket 83 43264272 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:36,316 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:39:36,316 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:36,317 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:36,317 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 43284016 2017-10-11 17:39:36,317 | TRACE | remote.py ( 86) | connect | connecting socket 84 43139728 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:36,317 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:39:36,317 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:36,318 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:36,318 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 81 43065120 2017-10-11 17:39:36,318 | TRACE | remote.py ( 86) | connect | connecting socket 86 43319616 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:36,318 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:39:36,318 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:36,319 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:36,319 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 82 43210704 2017-10-11 17:39:36,319 | TRACE | remote.py ( 86) | connect | connecting socket 91 43247424 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:36,319 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:39:36,321 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:39:36,335 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:39:36,338 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:39:36,338 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:39:36,338 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:39:36,338 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:39:36,338 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:39:36,339 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:39:36,339 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:39:36,688 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 178 2017-10-11 17:39:36,688 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:36,688 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 181 to run in 10 seconds 2017-10-11 17:39:46,691 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 181 2017-10-11 17:39:46,692 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:46,692 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 182 to run in 10 seconds 2017-10-11 17:39:51,317 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:39:51,318 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:39:51,318 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:39:51,318 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:39:51,318 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:39:51,319 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:39:51,319 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:39:51,320 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:39:51,321 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:39:51,333 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:39:51,335 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:39:51,336 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:39:51,336 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:39:51,337 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:39:51,337 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:39:51,337 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:39:51,337 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:39:56,527 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:39:56,528 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node3 2017-10-11 17:39:56,528 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node3 2017-10-11 17:39:56,528 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node4 2017-10-11 17:39:56,528 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node4 2017-10-11 17:39:56,528 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node2 2017-10-11 17:39:56,529 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node2 2017-10-11 17:39:56,529 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node2 2017-10-11 17:39:56,529 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node2 2017-10-11 17:39:56,529 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node3 2017-10-11 17:39:56,529 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node3 2017-10-11 17:39:56,530 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node4 2017-10-11 17:39:56,530 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node4 2017-10-11 17:39:56,553 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:39:56,553 | DEBUG | zstack.py ( 663) | handlePingPong | Node5C got ping from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:39:56,553 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'po' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:39:56,553 | DEBUG | zstack.py ( 645) | sendPingPong | Node5C ponged b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:39:56,565 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:39:56,566 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node1 2017-10-11 17:39:56,566 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node1 2017-10-11 17:39:56,567 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":16,"ppSeqNo":5}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,568 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":16,"ppSeqNo":5}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,571 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":16,"ppSeqNo":5}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,572 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,572 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4}, 'Node4') 2017-10-11 17:39:56,572 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,574 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,575 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node4') 2017-10-11 17:39:56,575 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,576 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,576 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 5, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'Node4') 2017-10-11 17:39:56,576 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,576 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,576 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node4') 2017-10-11 17:39:56,577 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,577 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":16,"ppSeqNo":5,"ledgerId":1,"viewNo":4,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,577 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":16,"ppSeqNo":5,"ledgerId":1,"viewNo":4,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,577 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":16,"ppSeqNo":5,"ledgerId":1,"viewNo":4,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,589 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,589 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4}, 'Node2') 2017-10-11 17:39:56,589 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,589 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,589 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node2') 2017-10-11 17:39:56,590 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,590 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,590 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 5, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'Node2') 2017-10-11 17:39:56,590 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,590 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,590 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node2') 2017-10-11 17:39:56,591 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,591 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} from Node4 2017-10-11 17:39:56,591 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node5 stashing a message for a future view: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:39:56,591 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node4 2017-10-11 17:39:56,592 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,592 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,592 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,593 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:56,593 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:56,593 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,593 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,594 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} from Node2 2017-10-11 17:39:56,594 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node5 stashing a message for a future view: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:39:56,594 | INFO | node.py (1967) | _start_view_change_if_possible | VIEW CHANGE: Node5 starting view change for 4 after 2 view change indications from other nodes 2017-10-11 17:39:56,594 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action _check_view_change_completed with id 183 to run in 60 seconds 2017-10-11 17:39:56,594 | DEBUG | replica.py ( 455) | on_view_change_start | Node5:0 setting last prepared for master to None 2017-10-11 17:39:56,594 | DEBUG | node.py (2146) | startViewChange | Node5 resetting monitor stats after view change 2017-10-11 17:39:56,594 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,595 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}, 'Node4') 2017-10-11 17:39:56,602 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}, 'Node2') 2017-10-11 17:39:56,602 | DEBUG | node.py ( 996) | processStashedMsgsForView | Node5 processed 2 stashed msgs for view no 4 2017-10-11 17:39:56,603 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node5 changed to view 4, will start catchup now 2017-10-11 17:39:56,604 | DEBUG | node.py (2185) | start_catchup | Node5 does not start the catchup procedure because it is already in this state 2017-10-11 17:39:56,604 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node2 2017-10-11 17:39:56,604 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,604 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,604 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,606 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:56,606 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:56,607 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,607 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,607 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:39:56,607 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,608 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,608 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:39:56,608 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,608 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,609 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,609 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:39:56,610 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node2 2017-10-11 17:39:56,610 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,621 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node2 2017-10-11 17:39:56,621 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node4 2017-10-11 17:39:56,622 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,622 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node4 2017-10-11 17:39:56,622 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node1 2017-10-11 17:39:56,622 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node1 2017-10-11 17:39:56,622 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,623 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":16,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"ppSeqNo":5}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,623 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":16,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"ppSeqNo":5}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,629 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":16,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"ppSeqNo":5}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,630 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,630 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4}, 'Node3') 2017-10-11 17:39:56,630 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,630 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,630 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node3') 2017-10-11 17:39:56,631 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,631 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,631 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 5, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'Node3') 2017-10-11 17:39:56,631 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,631 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,632 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node3') 2017-10-11 17:39:56,632 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,632 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} from Node3 2017-10-11 17:39:56,632 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}, 'Node3') 2017-10-11 17:39:56,635 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node3 2017-10-11 17:39:56,637 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,638 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,638 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,639 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:56,641 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:56,641 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,641 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,642 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node3 2017-10-11 17:39:56,642 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,642 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":4,"viewNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node3 2017-10-11 17:39:56,643 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,643 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,644 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4}, 'Node2') 2017-10-11 17:39:56,644 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,644 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} from Node2 2017-10-11 17:39:56,645 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node5 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-10-11 17:39:56,657 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:39:56,658 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,659 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4}, 'Node4') 2017-10-11 17:39:56,659 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,659 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":16,"ppSeqNo":5,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,659 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":16,"ppSeqNo":5,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:56,659 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":16,"ppSeqNo":5,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,659 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,660 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4}, 'Node1') 2017-10-11 17:39:56,660 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} 2017-10-11 17:39:56,660 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,660 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node1') 2017-10-11 17:39:56,660 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} 2017-10-11 17:39:56,660 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,661 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 5, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'Node1') 2017-10-11 17:39:56,661 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} 2017-10-11 17:39:56,662 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,662 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node1') 2017-10-11 17:39:56,662 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} 2017-10-11 17:39:56,662 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} from Node4 2017-10-11 17:39:56,663 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_CPs_if_needed) with id 1 to run in 20 seconds 2017-10-11 17:39:56,663 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node5 deciding on the basis of CPs {'Node2': CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4}, 'Node4': CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4}} and f 1 2017-10-11 17:39:56,671 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:56,671 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,671 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,672 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,672 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,672 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,672 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,672 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 0 2017-10-11 17:39:56,672 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node5 started catching up with consistency proof CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ('4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H',), 'seqNoStart': 4} 2017-10-11 17:39:56,673 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 0, 'seqNoEnd': 5, 'catchupTill': 5, 'seqNoStart': 5} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,673 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_txns_if_needed) with id 2 to run in 5.1 seconds 2017-10-11 17:39:56,673 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}], 'viewNo': 4} from Node1 2017-10-11 17:39:56,673 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4}, 'Node1') 2017-10-11 17:39:56,674 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node1 2017-10-11 17:39:56,674 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,682 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 4 with 5 2017-10-11 17:39:56,682 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:56,683 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:56,683 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,683 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,684 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"CATCHUP_REQ","ledgerId":0,"seqNoEnd":5,"catchupTill":5,"seqNoStart":5}' to Node1 2017-10-11 17:39:56,684 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,684 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"CATCHUP_REQ","ledgerId":0,"seqNoEnd":5,"catchupTill":5,"seqNoStart":5}' to Node1 2017-10-11 17:39:56,684 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,685 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,685 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4}, 'Node3') 2017-10-11 17:39:56,685 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} 2017-10-11 17:39:56,685 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ppSeqNo': 6, 'oldMerkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'seqNoEnd': 5, 'viewNo': 4, 'hashes': ['4EnM7DdZQSDmgdTNcWHWVxn75W9pPMZtC1Gxjxk4TP7H'], 'seqNoStart': 4} from Node3 2017-10-11 17:39:56,686 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,698 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 182 2017-10-11 17:39:56,698 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:39:56,698 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 184 to run in 10 seconds 2017-10-11 17:39:56,699 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,699 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CATCHUP_REP{'consProof': [], 'ledgerId': 0, 'txns': {'5': {'reqId': 1507743596333992, 'data': {'node_ip': '10.0.0.6', 'alias': 'Node5', 'client_ip': '10.0.0.6', 'client_port': 9702, 'node_port': 9701, 'services': ['VALIDATOR']}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1507743596, 'type': '0', 'signature': '53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4'}}} 2017-10-11 17:39:56,699 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': [], 'ledgerId': 0, 'op': 'CATCHUP_REP', 'txns': {'5': {'reqId': 1507743596333992, 'data': {'node_ip': '10.0.0.6', 'alias': 'Node5', 'client_ip': '10.0.0.6', 'client_port': 9702, 'node_port': 9701, 'services': ['VALIDATOR']}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1507743596, 'type': '0', 'signature': '53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4'}}}, 'Node1') 2017-10-11 17:39:56,700 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': [], 'ledgerId': 0, 'txns': {'5': {'reqId': 1507743596333992, 'data': {'node_ip': '10.0.0.6', 'alias': 'Node5', 'client_ip': '10.0.0.6', 'client_port': 9702, 'node_port': 9701, 'services': ['VALIDATOR']}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1507743596, 'type': '0', 'signature': '53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4'}}} 2017-10-11 17:39:56,700 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node1: CATCHUP_REP{'consProof': [], 'ledgerId': 0, 'txns': {'5': {'reqId': 1507743596333992, 'data': {'node_ip': '10.0.0.6', 'alias': 'Node5', 'client_ip': '10.0.0.6', 'client_port': 9702, 'node_port': 9701, 'services': ['VALIDATOR']}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1507743596, 'type': '0', 'signature': '53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4'}}} 2017-10-11 17:39:56,700 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 1 transactions in the catchup from Node1 2017-10-11 17:39:56,700 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,700 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 1 of them now, from 5 to 5 2017-10-11 17:39:56,701 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 5, 5, b'&-\xc60\x157\xc2\xe0\x9d*|\xc0N\x96\xc0\x0eV}\xa7\x12\x0f\x92\x8e\xa6\x16+\x1d80\x13\xaa\x10', b'&-\xc60\x157\xc2\xe0\x9d*|\xc0N\x96\xc0\x0eV}\xa7\x12\x0f\x92\x8e\xa6\x16+\x1d80\x13\xaa\x10', [] 2017-10-11 17:39:56,702 | DEBUG | pool_manager.py ( 309) | nodeServicesChanged | Node 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc not changing services since it is same as existing 2017-10-11 17:39:56,702 | DEBUG | node.py ( 879) | sendPoolInfoToClients | Node5 sending new node info {'reqId': 1507743596333992, 'data': {'node_ip': '10.0.0.6', 'alias': 'Node5', 'client_ip': '10.0.0.6', 'client_port': 9702, 'node_port': 9701, 'services': ['VALIDATOR']}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'txnTime': 1507743596, 'seqNo': 5, 'type': '0', 'signature': '53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4'} to all clients 2017-10-11 17:39:56,703 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"op":"POOL_LEDGER_TXNS","txn":{"reqId":1507743596333992,"data":{"node_ip":"10.0.0.6","alias":"Node5","client_ip":"10.0.0.6","client_port":9702,"node_port":9701,"services":["VALIDATOR"]},"identifier":"XhYtvJqezMUKfF6KVNaGmT","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","txnTime":1507743596,"seqNo":5,"type":"0","signature":"53ucmoX7GrcbUXergYoZCWcJ6kPwgify3bStZkVwKo58dPKmGPZncmtFhdmqiHkqYiruLn6FVorn5AWWswmmYmB4"}}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:39:56,704 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 1 catchup replies with sequence numbers [5] 2017-10-11 17:39:56,704 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 1 and nodes set() 2017-10-11 17:39:56,705 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,705 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 2 2017-10-11 17:39:56,705 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,705 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 2 2017-10-11 17:39:56,705 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,706 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 2 2017-10-11 17:39:56,706 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,706 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 2 2017-10-11 17:39:56,706 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:39:56,706 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 2 2017-10-11 17:39:56,706 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 4 stashed ledger statuses for ledger 2 2017-10-11 17:39:56,706 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,707 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,707 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,707 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,707 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,708 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,708 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,709 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,709 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,709 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,709 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,709 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,710 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,710 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 0, caught up 1 in total 2017-10-11 17:39:56,710 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node2 into one transmission 2017-10-11 17:39:56,710 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-10-11 17:39:56,711 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' 2017-10-11 17:39:56,711 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' to Node2 2017-10-11 17:39:56,711 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,711 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node3 into one transmission 2017-10-11 17:39:56,711 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-10-11 17:39:56,712 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' 2017-10-11 17:39:56,712 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' to Node3 2017-10-11 17:39:56,712 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,712 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node4 into one transmission 2017-10-11 17:39:56,713 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-10-11 17:39:56,713 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' 2017-10-11 17:39:56,713 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' to Node4 2017-10-11 17:39:56,713 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,713 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node1 into one transmission 2017-10-11 17:39:56,714 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-10-11 17:39:56,714 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' 2017-10-11 17:39:56,714 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' to Node1 2017-10-11 17:39:56,714 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,727 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:39:56,728 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}'], 'signature': None} 2017-10-11 17:39:56,728 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,728 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}'], 'signature': None} 2017-10-11 17:39:56,728 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,729 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node2') 2017-10-11 17:39:56,729 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,729 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,729 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node2') 2017-10-11 17:39:56,729 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,730 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,730 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node2') 2017-10-11 17:39:56,730 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,730 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,730 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,730 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}'], 'signature': None} 2017-10-11 17:39:56,731 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,731 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:39:56,731 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,731 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,731 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:39:56,731 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,732 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,732 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node4') 2017-10-11 17:39:56,732 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,732 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":2,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":2,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoEnd":11,"seqNoStart":0}'], 'signature': None} 2017-10-11 17:39:56,733 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":2,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":2,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoEnd":11,"seqNoStart":0}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,733 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":2,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":2,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoEnd":11,"seqNoStart":0}'], 'signature': None} 2017-10-11 17:39:56,733 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,733 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:39:56,735 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,736 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,736 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:39:56,736 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,738 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,739 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node3') 2017-10-11 17:39:56,739 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,739 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,740 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,740 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,740 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,740 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,740 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,741 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,741 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,741 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node2 2017-10-11 17:39:56,741 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node5 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-10-11 17:39:56,742 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,742 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,742 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,742 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,743 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,743 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,743 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,743 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,744 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node4 2017-10-11 17:39:56,745 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_CPs_if_needed) with id 3 to run in 20 seconds 2017-10-11 17:39:56,745 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node5 deciding on the basis of CPs {'Node2': CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node4': CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}} and f 1 2017-10-11 17:39:56,745 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:56,745 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,745 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,746 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,746 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,746 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,746 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,746 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 2 2017-10-11 17:39:56,746 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node5 started catching up with consistency proof CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ('dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F',), 'seqNoStart': 0} 2017-10-11 17:39:56,747 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 2, 'seqNoEnd': 3, 'catchupTill': 11, 'seqNoStart': 1} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,747 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 2, 'seqNoEnd': 6, 'catchupTill': 11, 'seqNoStart': 4} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,747 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 2, 'seqNoEnd': 9, 'catchupTill': 11, 'seqNoStart': 7} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,747 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 2, 'seqNoEnd': 11, 'catchupTill': 11, 'seqNoStart': 10} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,748 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_txns_if_needed) with id 4 to run in 21.2 seconds 2017-10-11 17:39:56,748 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,748 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,748 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,750 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,751 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,751 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,751 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node3 2017-10-11 17:39:56,751 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,752 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node2 into one transmission 2017-10-11 17:39:56,752 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}', b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":6,"catchupTill":11,"seqNoStart":4}']) 2017-10-11 17:39:56,752 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":2,\\"seqNoEnd\\":6,\\"catchupTill\\":11,\\"seqNoStart\\":4}"],"signature":null}' 2017-10-11 17:39:56,752 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":2,\\"seqNoEnd\\":6,\\"catchupTill\\":11,\\"seqNoStart\\":4}"],"signature":null}' to Node2 2017-10-11 17:39:56,752 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,753 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":9,"catchupTill":11,"seqNoStart":7}' to Node3 2017-10-11 17:39:56,753 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,753 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":9,"catchupTill":11,"seqNoStart":7}' to Node3 2017-10-11 17:39:56,753 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node4 into one transmission 2017-10-11 17:39:56,753 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":0,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}', b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":11,"catchupTill":11,"seqNoStart":10}']) 2017-10-11 17:39:56,753 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":2,\\"seqNoEnd\\":11,\\"catchupTill\\":11,\\"seqNoStart\\":10}"],"signature":null}' 2017-10-11 17:39:56,753 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":2,\\"seqNoEnd\\":11,\\"catchupTill\\":11,\\"seqNoStart\\":10}"],"signature":null}' to Node4 2017-10-11 17:39:56,754 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,754 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":3,"catchupTill":11,"seqNoStart":1}' to Node1 2017-10-11 17:39:56,754 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,754 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"CATCHUP_REQ","ledgerId":2,"seqNoEnd":3,"catchupTill":11,"seqNoStart":1}' to Node1 2017-10-11 17:39:56,755 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,755 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":2,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"ledgerId":2,"seqNoEnd":11,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF"}'], 'signature': None} 2017-10-11 17:39:56,755 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":2,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"ledgerId":2,"seqNoEnd":11,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:56,755 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":2,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"ledgerId":2,"seqNoEnd":11,"oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF"}'], 'signature': None} 2017-10-11 17:39:56,756 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,756 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:39:56,756 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,756 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,756 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:39:56,756 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,757 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,757 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'op': 'CONSISTENCY_PROOF', 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node1') 2017-10-11 17:39:56,757 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,757 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,758 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,759 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 0 with 11 2017-10-11 17:39:56,759 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node1 2017-10-11 17:39:56,759 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,760 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,763 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CATCHUP_REP{'consProof': ['FFusKBPJVV3mVuKVMNsZ1KcJ77B3EPoyMZoXiPdLTKdv', 'A2gFxw6PhcSoinTiGfzWmSNYPeL1MQuAn9DnkTnFrxjw', '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', '7buRcGpwMpZCtkGZEuKX92Z8ffM2kMHm3aPgkYQuzdhS', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'2': {'reqId': 1507741607555952, 'action': 'start', 'force': False, 'name': 'upgrade40', 'justification': None, 'txnTime': 1507741607, 'type': '109', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.40', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}, 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}, '3': {'reqId': 1507741799688892, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741799, 'type': '110', 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, '1': {'reqId': 1507741413733229, 'action': 'start', 'force': True, 'name': 'upgrade37', 'justification': None, 'txnTime': 1507741413, 'type': '109', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.37', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}} 2017-10-11 17:39:56,763 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['FFusKBPJVV3mVuKVMNsZ1KcJ77B3EPoyMZoXiPdLTKdv', 'A2gFxw6PhcSoinTiGfzWmSNYPeL1MQuAn9DnkTnFrxjw', '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', '7buRcGpwMpZCtkGZEuKX92Z8ffM2kMHm3aPgkYQuzdhS', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'op': 'CATCHUP_REP', 'txns': {'2': {'reqId': 1507741607555952, 'action': 'start', 'force': False, 'name': 'upgrade40', 'justification': None, 'txnTime': 1507741607, 'type': '109', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.40', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}, 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}, '3': {'reqId': 1507741799688892, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741799, 'type': '110', 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, '1': {'reqId': 1507741413733229, 'action': 'start', 'force': True, 'name': 'upgrade37', 'justification': None, 'txnTime': 1507741413, 'type': '109', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.37', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}}, 'Node1') 2017-10-11 17:39:56,764 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['FFusKBPJVV3mVuKVMNsZ1KcJ77B3EPoyMZoXiPdLTKdv', 'A2gFxw6PhcSoinTiGfzWmSNYPeL1MQuAn9DnkTnFrxjw', '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', '7buRcGpwMpZCtkGZEuKX92Z8ffM2kMHm3aPgkYQuzdhS', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'2': {'reqId': 1507741607555952, 'action': 'start', 'force': False, 'name': 'upgrade40', 'justification': None, 'txnTime': 1507741607, 'type': '109', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.40', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}, 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}, '3': {'reqId': 1507741799688892, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741799, 'type': '110', 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, '1': {'reqId': 1507741413733229, 'action': 'start', 'force': True, 'name': 'upgrade37', 'justification': None, 'txnTime': 1507741413, 'type': '109', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.37', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}} 2017-10-11 17:39:56,764 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node1: CATCHUP_REP{'consProof': ['FFusKBPJVV3mVuKVMNsZ1KcJ77B3EPoyMZoXiPdLTKdv', 'A2gFxw6PhcSoinTiGfzWmSNYPeL1MQuAn9DnkTnFrxjw', '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', '7buRcGpwMpZCtkGZEuKX92Z8ffM2kMHm3aPgkYQuzdhS', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'2': {'reqId': 1507741607555952, 'action': 'start', 'force': False, 'name': 'upgrade40', 'justification': None, 'txnTime': 1507741607, 'type': '109', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.40', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}, 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}, '3': {'reqId': 1507741799688892, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741799, 'type': '110', 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, '1': {'reqId': 1507741413733229, 'action': 'start', 'force': True, 'name': 'upgrade37', 'justification': None, 'txnTime': 1507741413, 'type': '109', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'timeout': 10, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'version': '1.1.37', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}} 2017-10-11 17:39:56,764 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 3 transactions in the catchup from Node1 2017-10-11 17:39:56,764 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,764 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 3 of them now, from 1 to 3 2017-10-11 17:39:56,765 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 3, 11, b'Yk%;\x90\x95\xef\xbd-G~o\xd8F\x9a\xcd0\x0c\xbe\xde6\xc1\xbf^\xddQ\x9ey\x87\xc8\x0c\xf8', b'\tT,\xe2\xff\x1e\xe5\xf6\x87\x1d\x8f\x8c4\xeaE\xcb{\x8e\xd0i\xd6g\x805\xbb\xc5\x03\x17\x07\x92J\x98', [b'\xd3\xd5\x86\x96\xc7\xee\xc2%\xa9\xc8]\xdc\r\x90D\xd2WZ\xd3\x08\xf8\x07\xabY\x8c?\x01\xfbp\x00\x01\xa5', b'\x86\'&\xfe"\xa7C\xa7dA\x1a\xf0\x9e\xa0\xb3\x85k\xf4\xda\xbd\xe4z\xa4s\xb6{\xb9\xa7\xa7^%\x1e', b'0\x1e\xae}*\x84C\x94\xff\xff\x84\x06\xbb\x0f\x8c\xa6X\x7f\x8924\x1e\x8b\xb3\xd2$\x00%\xe2Pf\xc4', b'b\x17%8\x1a\xd0\xbf^\xbbje\xa6Y\x92\x97\x9a\xa1\x17o\x9e\xbc_\x12z\x810\x99a\x06=\x8a\x01', b'\xd0\x8a\x99n^\x91\x99\x1a\xa1\xfc\x0e\x84\x08zM\xa4[s\xea\xe8%]\xfaP\xa3&\x005v\xfd\x86\xde'] 2017-10-11 17:39:56,769 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 3 catchup replies with sequence numbers [1, 2, 3] 2017-10-11 17:39:56,770 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:39:56,771 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CATCHUP_REP{'consProof': ['2wtwpQvNXd9n6F3tPknDwhb9i65kHdgqSz6pxUowWna9', '8WCg3b5eB4mYZm75CpAcjzWUXdpE1G3WfDdQzc9D1Sbe', 'xCWRiz1zkjkisTckxKM5Npe3XGWs2JaQkWfJix4Y41B', 'BdrMQeQXwNnWEbCSyx63eTxNK2LPnvZNtx5LzJYAf7Gh'], 'ledgerId': 2, 'txns': {'8': {'reqId': 1507742419972772, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '7': {'reqId': 1507742399505818, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742430, 'type': '110', 'signature': '4rxkKqd84RHgfbdZT64aW2rUhVRjED86ADqmCL1wimioLPpChACaZ7wGYuYcE5WLE3eYNPFWKv18SBSgapMXNjUL'}, '9': {'reqId': 1507742700019586, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742700, 'type': '110', 'signature': '3j1QCNHT5Ef5eKBQnSmy1ZUV6tvTzX2p3LDmgcD6SjmBx1NLtrZZB3eeHbTUwC5yLLeDbt2Wfc4Yg7cpbYNzELyu'}}} 2017-10-11 17:39:56,771 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['2wtwpQvNXd9n6F3tPknDwhb9i65kHdgqSz6pxUowWna9', '8WCg3b5eB4mYZm75CpAcjzWUXdpE1G3WfDdQzc9D1Sbe', 'xCWRiz1zkjkisTckxKM5Npe3XGWs2JaQkWfJix4Y41B', 'BdrMQeQXwNnWEbCSyx63eTxNK2LPnvZNtx5LzJYAf7Gh'], 'ledgerId': 2, 'op': 'CATCHUP_REP', 'txns': {'8': {'reqId': 1507742419972772, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '7': {'reqId': 1507742399505818, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742430, 'type': '110', 'signature': '4rxkKqd84RHgfbdZT64aW2rUhVRjED86ADqmCL1wimioLPpChACaZ7wGYuYcE5WLE3eYNPFWKv18SBSgapMXNjUL'}, '9': {'reqId': 1507742700019586, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742700, 'type': '110', 'signature': '3j1QCNHT5Ef5eKBQnSmy1ZUV6tvTzX2p3LDmgcD6SjmBx1NLtrZZB3eeHbTUwC5yLLeDbt2Wfc4Yg7cpbYNzELyu'}}}, 'Node3') 2017-10-11 17:39:56,772 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['2wtwpQvNXd9n6F3tPknDwhb9i65kHdgqSz6pxUowWna9', '8WCg3b5eB4mYZm75CpAcjzWUXdpE1G3WfDdQzc9D1Sbe', 'xCWRiz1zkjkisTckxKM5Npe3XGWs2JaQkWfJix4Y41B', 'BdrMQeQXwNnWEbCSyx63eTxNK2LPnvZNtx5LzJYAf7Gh'], 'ledgerId': 2, 'txns': {'8': {'reqId': 1507742419972772, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '7': {'reqId': 1507742399505818, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742430, 'type': '110', 'signature': '4rxkKqd84RHgfbdZT64aW2rUhVRjED86ADqmCL1wimioLPpChACaZ7wGYuYcE5WLE3eYNPFWKv18SBSgapMXNjUL'}, '9': {'reqId': 1507742700019586, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742700, 'type': '110', 'signature': '3j1QCNHT5Ef5eKBQnSmy1ZUV6tvTzX2p3LDmgcD6SjmBx1NLtrZZB3eeHbTUwC5yLLeDbt2Wfc4Yg7cpbYNzELyu'}}} 2017-10-11 17:39:56,772 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"consProof":[],"txns":{"10":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","type":"110","signature":"4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s","data":{"action":"complete","version":"1.1.40"},"reqId":1507742719661587,"txnTime":1507742719},"11":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","type":"110","signature":"2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT","data":{"action":"complete","version":"1.1.40"},"reqId":1507742729130945,"txnTime":1507742729}},"op":"CATCHUP_REP","ledgerId":2}'], 'signature': None} 2017-10-11 17:39:56,772 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"consProof":[],"txns":{"10":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","type":"110","signature":"4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s","data":{"action":"complete","version":"1.1.40"},"reqId":1507742719661587,"txnTime":1507742719},"11":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","type":"110","signature":"2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT","data":{"action":"complete","version":"1.1.40"},"reqId":1507742729130945,"txnTime":1507742729}},"op":"CATCHUP_REP","ledgerId":2}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,772 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"seqNoEnd":11,"ledgerId":2,"viewNo":4,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"seqNoStart":0,"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2}', '{"consProof":[],"txns":{"10":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","type":"110","signature":"4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s","data":{"action":"complete","version":"1.1.40"},"reqId":1507742719661587,"txnTime":1507742719},"11":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","type":"110","signature":"2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT","data":{"action":"complete","version":"1.1.40"},"reqId":1507742729130945,"txnTime":1507742729}},"op":"CATCHUP_REP","ledgerId":2}'], 'signature': None} 2017-10-11 17:39:56,772 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,773 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node4') 2017-10-11 17:39:56,773 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,773 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,773 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node4') 2017-10-11 17:39:56,773 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,773 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CATCHUP_REP{'consProof': [], 'ledgerId': 2, 'txns': {'10': {'reqId': 1507742719661587, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742719, 'type': '110', 'signature': '4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s'}, '11': {'reqId': 1507742729130945, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742729, 'type': '110', 'signature': '2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT'}}} 2017-10-11 17:39:56,774 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': [], 'op': 'CATCHUP_REP', 'ledgerId': 2, 'txns': {'10': {'reqId': 1507742719661587, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742719, 'type': '110', 'signature': '4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s'}, '11': {'reqId': 1507742729130945, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742729, 'type': '110', 'signature': '2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT'}}}, 'Node4') 2017-10-11 17:39:56,774 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': [], 'ledgerId': 2, 'txns': {'10': {'reqId': 1507742719661587, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742719, 'type': '110', 'signature': '4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s'}, '11': {'reqId': 1507742729130945, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742729, 'type': '110', 'signature': '2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT'}}} 2017-10-11 17:39:56,774 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"consProof":["Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh","Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e","Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5"],"ledgerId":2,"txns":{"4":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ","txnTime":1507741828,"reqId":1507741828468646,"data":{"action":"complete","version":"1.1.40"},"type":"110"},"5":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS","txnTime":1507742099,"reqId":1507742099432084,"data":{"action":"in_progress","version":"1.1.40"},"type":"110"},"6":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369","txnTime":1507742430,"reqId":1507742128555022,"data":{"action":"complete","version":"1.1.40"},"type":"110"}},"op":"CATCHUP_REP"}'], 'signature': None} 2017-10-11 17:39:56,774 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"consProof":["Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh","Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e","Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5"],"ledgerId":2,"txns":{"4":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ","txnTime":1507741828,"reqId":1507741828468646,"data":{"action":"complete","version":"1.1.40"},"type":"110"},"5":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS","txnTime":1507742099,"reqId":1507742099432084,"data":{"action":"in_progress","version":"1.1.40"},"type":"110"},"6":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369","txnTime":1507742430,"reqId":1507742128555022,"data":{"action":"complete","version":"1.1.40"},"type":"110"}},"op":"CATCHUP_REP"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,774 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"newMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"seqNoEnd":11,"viewNo":4,"ppSeqNo":2,"seqNoStart":0,"hashes":["dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', '{"consProof":["Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh","Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e","Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5"],"ledgerId":2,"txns":{"4":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ","txnTime":1507741828,"reqId":1507741828468646,"data":{"action":"complete","version":"1.1.40"},"type":"110"},"5":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS","txnTime":1507742099,"reqId":1507742099432084,"data":{"action":"in_progress","version":"1.1.40"},"type":"110"},"6":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369","txnTime":1507742430,"reqId":1507742128555022,"data":{"action":"complete","version":"1.1.40"},"type":"110"}},"op":"CATCHUP_REP"}'], 'signature': None} 2017-10-11 17:39:56,775 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,775 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node2') 2017-10-11 17:39:56,775 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,775 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,775 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0}, 'Node2') 2017-10-11 17:39:56,776 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} 2017-10-11 17:39:56,776 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CATCHUP_REP{'consProof': ['Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh', 'Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e', 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'4': {'reqId': 1507741828468646, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741828, 'type': '110', 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, '6': {'reqId': 1507742128555022, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '5': {'reqId': 1507742099432084, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742099, 'type': '110', 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}}} 2017-10-11 17:39:56,776 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh', 'Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e', 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'op': 'CATCHUP_REP', 'txns': {'4': {'reqId': 1507741828468646, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741828, 'type': '110', 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, '6': {'reqId': 1507742128555022, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '5': {'reqId': 1507742099432084, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742099, 'type': '110', 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}}}, 'Node2') 2017-10-11 17:39:56,776 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh', 'Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e', 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'4': {'reqId': 1507741828468646, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741828, 'type': '110', 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, '6': {'reqId': 1507742128555022, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '5': {'reqId': 1507742099432084, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742099, 'type': '110', 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}}} 2017-10-11 17:39:56,776 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node3: CATCHUP_REP{'consProof': ['2wtwpQvNXd9n6F3tPknDwhb9i65kHdgqSz6pxUowWna9', '8WCg3b5eB4mYZm75CpAcjzWUXdpE1G3WfDdQzc9D1Sbe', 'xCWRiz1zkjkisTckxKM5Npe3XGWs2JaQkWfJix4Y41B', 'BdrMQeQXwNnWEbCSyx63eTxNK2LPnvZNtx5LzJYAf7Gh'], 'ledgerId': 2, 'txns': {'8': {'reqId': 1507742419972772, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '7': {'reqId': 1507742399505818, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742430, 'type': '110', 'signature': '4rxkKqd84RHgfbdZT64aW2rUhVRjED86ADqmCL1wimioLPpChACaZ7wGYuYcE5WLE3eYNPFWKv18SBSgapMXNjUL'}, '9': {'reqId': 1507742700019586, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742700, 'type': '110', 'signature': '3j1QCNHT5Ef5eKBQnSmy1ZUV6tvTzX2p3LDmgcD6SjmBx1NLtrZZB3eeHbTUwC5yLLeDbt2Wfc4Yg7cpbYNzELyu'}}} 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 3 transactions in the catchup from Node3 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 3 of them now, from 7 to 9 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 0 catchup replies with sequence numbers [] 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node4 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,777 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node4 2017-10-11 17:39:56,778 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,778 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node4: CATCHUP_REP{'consProof': [], 'ledgerId': 2, 'txns': {'10': {'reqId': 1507742719661587, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'txnTime': 1507742719, 'type': '110', 'signature': '4LiVfEqMrdpAMc88tt7dSDBwL5bYM5FDu7bfh3ehB136bKmYWYn5hNSrBxmTnqC95FTmescMhzhYeTKmA2htQE3s'}, '11': {'reqId': 1507742729130945, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'txnTime': 1507742729, 'type': '110', 'signature': '2fpeaZb1WqzADvsZEwXmPR2PWuzQbgv2WXV6WX8rzeMCi8XUQmDW2pM6dufFfJRrCsZffiv3Yz3K5S6Taz7UG4PT'}}} 2017-10-11 17:39:56,778 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 2 transactions in the catchup from Node4 2017-10-11 17:39:56,778 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,780 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 5 of them now, from 7 to 11 2017-10-11 17:39:56,780 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 0 catchup replies with sequence numbers [] 2017-10-11 17:39:56,780 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node2 2017-10-11 17:39:56,781 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,781 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ppSeqNo': 2, 'oldMerkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'seqNoEnd': 11, 'viewNo': 4, 'hashes': ['dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'], 'seqNoStart': 0} from Node2 2017-10-11 17:39:56,781 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,781 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node2: CATCHUP_REP{'consProof': ['Fh2i6sxtnC5UiuxyEjCyPFi8CcE4XKAXf4G3HFtdiGgh', 'Du8Hv1EZrkFw4DH64JwdzyqFXpZsesfsfdeQsctTUU6e', 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'F34NzGhmUH2FAx4QyqZR9NMimjGWK7sKEnCSSzXCSqv5'], 'ledgerId': 2, 'txns': {'4': {'reqId': 1507741828468646, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'txnTime': 1507741828, 'type': '110', 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, '6': {'reqId': 1507742128555022, 'data': {'action': 'complete', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742430, 'type': '110', 'signature': 'ELTzNudpv66L4uqK8N2eD9V9b8zRZhaFox8fuxmXBn6n5DvALnFYFySyao2eWFzt7E7pKXKYqKPXF3rT41bk369'}, '5': {'reqId': 1507742099432084, 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'txnTime': 1507742099, 'type': '110', 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}}} 2017-10-11 17:39:56,781 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 3 transactions in the catchup from Node2 2017-10-11 17:39:56,782 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,782 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 8 of them now, from 4 to 11 2017-10-11 17:39:56,782 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 6, 11, b'K\xee\x83\x88S\x17\xc6\x93\x8e8!\xad\xe4\x8b\xad2\x9bY\xac\xe4_"\xb25,2\t\xa73\xc6n\xbf', b'\tT,\xe2\xff\x1e\xe5\xf6\x87\x1d\x8f\x8c4\xeaE\xcb{\x8e\xd0i\xd6g\x805\xbb\xc5\x03\x17\x07\x92J\x98', [b'\xdaD\xc8\x03g\xa8\xef\xd7!\xdeB2\xd4\xca=81*\xb5\x8d~\xa7\xcd_\x99\xcb\xf7@\xd8\x81\xe7"', b'\xbf\xa6\xb0\x99\xd4V\x03\xbf<\x9d=i\xbf&\xec+\x7f\xa8\xb0*\x1cx\xb0Q\xc0\xe5\xd8\xb8\x19\x8e\x89\x8b', b'\xca\xa7\xfa\xa4P\x8c\x1e/\xbcHU\x7f\xa6\xbb\x84,{9\n\xbf\x17%r\xb2\\sP\xd0re\xe5\x02', b'\xd0\x8a\x99n^\x91\x99\x1a\xa1\xfc\x0e\x84\x08zM\xa4[s\xea\xe8%]\xfaP\xa3&\x005v\xfd\x86\xde'] 2017-10-11 17:39:56,784 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 9, 11, b'x\xd6\x13WH\x07\xbb\xc7\xff\xfd\xb2\xe8\xd3\xa8\x9b\x06A\xcc\xf0+\xcdp\xb8<\xbb\x82\xbe\xc6\xdc\x83\x97\x1f', b'\tT,\xe2\xff\x1e\xe5\xf6\x87\x1d\x8f\x8c4\xeaE\xcb{\x8e\xd0i\xd6g\x805\xbb\xc5\x03\x17\x07\x92J\x98', [b'\x1c\xeb\xc9\x00G>I\x9e\xa1\x91\x95\x8f\xe6\xc1\x11\x03,(\x0bbB\x02\x1f,\x08?\x0e=(\xae8\xbe', b'o|\xd9\xd1\xcf\xf0\xdda\xcb\x80\xda\xc2\x9b\xf3\xc4\x8e processing config ledger for any POOL_CONFIGs 2017-10-11 17:39:56,787 | DEBUG | upgrader.py ( 208) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc processing config ledger for any upgrades 2017-10-11 17:39:56,787 | INFO | upgrader.py ( 214) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,787 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node5' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,787 | INFO | upgrader.py ( 304) | handleUpgradeTxn | Node 'Node5' disregards upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,788 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,788 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 1 2017-10-11 17:39:56,788 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,788 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 1 2017-10-11 17:39:56,788 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,788 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 1 2017-10-11 17:39:56,788 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,789 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 1 2017-10-11 17:39:56,789 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:39:56,789 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 1 2017-10-11 17:39:56,789 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 4 stashed ledger statuses for ledger 1 2017-10-11 17:39:56,789 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:56,789 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,789 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,790 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,790 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:56,790 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,790 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,792 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,792 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:56,792 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,792 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,793 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,793 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:56,793 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,793 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,793 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,794 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node5' last upgrade txn is None 2017-10-11 17:39:56,794 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 2, caught up 11 in total 2017-10-11 17:39:56,794 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node2 into one transmission 2017-10-11 17:39:56,795 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}']) 2017-10-11 17:39:56,795 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' 2017-10-11 17:39:56,795 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' to Node2 2017-10-11 17:39:56,795 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,795 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node3 into one transmission 2017-10-11 17:39:56,795 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}']) 2017-10-11 17:39:56,796 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' 2017-10-11 17:39:56,796 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' to Node3 2017-10-11 17:39:56,796 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,796 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node4 into one transmission 2017-10-11 17:39:56,796 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}']) 2017-10-11 17:39:56,796 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' 2017-10-11 17:39:56,797 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' to Node4 2017-10-11 17:39:56,797 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,797 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node1 into one transmission 2017-10-11 17:39:56,797 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}']) 2017-10-11 17:39:56,797 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' 2017-10-11 17:39:56,797 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}"],"signature":null}' to Node1 2017-10-11 17:39:56,797 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,813 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 4 messages through listener 2017-10-11 17:39:56,813 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}'], 'signature': None} 2017-10-11 17:39:56,814 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,814 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}'], 'signature': None} 2017-10-11 17:39:56,815 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,816 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:39:56,816 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,816 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,816 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:39:56,816 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,817 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,817 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node3') 2017-10-11 17:39:56,817 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,818 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}'], 'signature': None} 2017-10-11 17:39:56,818 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,818 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}'], 'signature': None} 2017-10-11 17:39:56,818 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,818 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:39:56,819 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,819 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,819 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:39:56,819 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,819 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,820 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node4') 2017-10-11 17:39:56,820 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,820 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"seqNoEnd":16,"viewNo":4,"ppSeqNo":5,"seqNoStart":9,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}'], 'signature': None} 2017-10-11 17:39:56,820 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"seqNoEnd":16,"viewNo":4,"ppSeqNo":5,"seqNoStart":9,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,820 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ledgerId":1,"seqNoEnd":16,"viewNo":4,"ppSeqNo":5,"seqNoStart":9,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}'], 'signature': None} 2017-10-11 17:39:56,820 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,821 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node2') 2017-10-11 17:39:56,821 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,821 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,821 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node2') 2017-10-11 17:39:56,821 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,822 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,822 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node2') 2017-10-11 17:39:56,822 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,822 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":5,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"ledgerId":1,"seqNoEnd":16,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF"}'], 'signature': None} 2017-10-11 17:39:56,822 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":5,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"ledgerId":1,"seqNoEnd":16,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:56,824 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"viewNo":4,"ppSeqNo":5,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"ledgerId":1,"seqNoEnd":16,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF"}'], 'signature': None} 2017-10-11 17:39:56,824 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,824 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:39:56,824 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,825 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,825 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:39:56,825 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:56,825 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,825 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'op': 'CONSISTENCY_PROOF', 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node1') 2017-10-11 17:39:56,825 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,826 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:56,827 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,827 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,827 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,828 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:56,828 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,828 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,828 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,829 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node3 2017-10-11 17:39:56,829 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node5 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-10-11 17:39:56,829 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:56,829 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,830 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,830 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,830 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:56,830 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,831 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,831 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,831 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node4 2017-10-11 17:39:56,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_CPs_if_needed) with id 5 to run in 20 seconds 2017-10-11 17:39:56,831 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node5 deciding on the basis of CPs {'Node3': CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node4': CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}} and f 1 2017-10-11 17:39:56,832 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:56,832 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,834 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,835 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,835 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,836 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,836 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,836 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 1 2017-10-11 17:39:56,836 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node5 started catching up with consistency proof CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ('jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'), 'seqNoStart': 9} 2017-10-11 17:39:56,836 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'seqNoEnd': 11, 'catchupTill': 16, 'seqNoStart': 10} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,837 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'seqNoEnd': 13, 'catchupTill': 16, 'seqNoStart': 12} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,837 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'seqNoEnd': 15, 'catchupTill': 16, 'seqNoStart': 14} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,837 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'seqNoEnd': 16, 'catchupTill': 16, 'seqNoStart': 16} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,837 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_txns_if_needed) with id 6 to run in 20.8 seconds 2017-10-11 17:39:56,838 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:56,838 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,838 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,838 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:56,838 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,839 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,839 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node2 2017-10-11 17:39:56,839 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,839 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 9 with 16 2017-10-11 17:39:56,840 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node1 2017-10-11 17:39:56,841 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,841 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":13,"catchupTill":16,"seqNoStart":12}' to Node2 2017-10-11 17:39:56,841 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,841 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":13,"catchupTill":16,"seqNoStart":12}' to Node2 2017-10-11 17:39:56,842 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node3 into one transmission 2017-10-11 17:39:56,842 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}', b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":15,"catchupTill":16,"seqNoStart":14}']) 2017-10-11 17:39:56,842 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":1,\\"seqNoEnd\\":15,\\"catchupTill\\":16,\\"seqNoStart\\":14}"],"signature":null}' 2017-10-11 17:39:56,842 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":1,\\"seqNoEnd\\":15,\\"catchupTill\\":16,\\"seqNoStart\\":14}"],"signature":null}' to Node3 2017-10-11 17:39:56,842 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,842 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node4 into one transmission 2017-10-11 17:39:56,842 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":9,"viewNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}', b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":16,"catchupTill":16,"seqNoStart":16}']) 2017-10-11 17:39:56,843 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":1,\\"seqNoEnd\\":16,\\"catchupTill\\":16,\\"seqNoStart\\":16}"],"signature":null}' 2017-10-11 17:39:56,843 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"op\\":\\"CATCHUP_REQ\\",\\"ledgerId\\":1,\\"seqNoEnd\\":16,\\"catchupTill\\":16,\\"seqNoStart\\":16}"],"signature":null}' to Node4 2017-10-11 17:39:56,843 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,843 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":11,"catchupTill":16,"seqNoStart":10}' to Node1 2017-10-11 17:39:56,843 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,843 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"CATCHUP_REQ","ledgerId":1,"seqNoEnd":11,"catchupTill":16,"seqNoStart":10}' to Node1 2017-10-11 17:39:56,850 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:39:56,851 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CATCHUP_REP{'consProof': ['FPVihNkFRp2M9DiCynmHpGY9i1DfTgvDB2fSmBPrTVzG', '7s2bfj7LTYYEmRdcCimvjsTHQBpZ9bo7PeCypXFuUHf2', '4jhsGUEJzC6HEwZJEUY5FqkTC8PSeRzPqw5d6khvq7x4', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'12': {'reqId': 1507741500981867, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741501, 'dest': '542MVr22zcHbVyGzaXmbT3', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU'}, '13': {'reqId': 1507741503995435, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741504, 'dest': '542MVr22zcHbVyGzaXmbT4', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ'}}} 2017-10-11 17:39:56,851 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['FPVihNkFRp2M9DiCynmHpGY9i1DfTgvDB2fSmBPrTVzG', '7s2bfj7LTYYEmRdcCimvjsTHQBpZ9bo7PeCypXFuUHf2', '4jhsGUEJzC6HEwZJEUY5FqkTC8PSeRzPqw5d6khvq7x4', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'op': 'CATCHUP_REP', 'txns': {'12': {'reqId': 1507741500981867, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741501, 'dest': '542MVr22zcHbVyGzaXmbT3', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU'}, '13': {'reqId': 1507741503995435, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741504, 'dest': '542MVr22zcHbVyGzaXmbT4', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ'}}}, 'Node2') 2017-10-11 17:39:56,851 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['FPVihNkFRp2M9DiCynmHpGY9i1DfTgvDB2fSmBPrTVzG', '7s2bfj7LTYYEmRdcCimvjsTHQBpZ9bo7PeCypXFuUHf2', '4jhsGUEJzC6HEwZJEUY5FqkTC8PSeRzPqw5d6khvq7x4', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'12': {'reqId': 1507741500981867, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741501, 'dest': '542MVr22zcHbVyGzaXmbT3', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU'}, '13': {'reqId': 1507741503995435, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741504, 'dest': '542MVr22zcHbVyGzaXmbT4', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ'}}} 2017-10-11 17:39:56,852 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"consProof":[],"txns":{"16":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","signature":"3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba","role":"2","verkey":"~RmCt3RtDvKDfQBSKxo4qvy","type":"1","reqId":1507743286863262,"txnTime":1507743286}},"op":"CATCHUP_REP","ledgerId":1}'], 'signature': None} 2017-10-11 17:39:56,852 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"consProof":[],"txns":{"16":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","signature":"3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba","role":"2","verkey":"~RmCt3RtDvKDfQBSKxo4qvy","type":"1","reqId":1507743286863262,"txnTime":1507743286}},"op":"CATCHUP_REP","ledgerId":1}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,852 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"seqNoEnd":16,"ledgerId":1,"viewNo":4,"hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"seqNoStart":9,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","op":"CONSISTENCY_PROOF","oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":5}', '{"consProof":[],"txns":{"16":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"XhYtvJqezMUKfF6KVNaGmT","signature":"3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba","role":"2","verkey":"~RmCt3RtDvKDfQBSKxo4qvy","type":"1","reqId":1507743286863262,"txnTime":1507743286}},"op":"CATCHUP_REP","ledgerId":1}'], 'signature': None} 2017-10-11 17:39:56,852 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,852 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node4') 2017-10-11 17:39:56,852 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,853 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,853 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node4') 2017-10-11 17:39:56,853 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,854 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CATCHUP_REP{'consProof': [], 'ledgerId': 1, 'txns': {'16': {'role': '2', 'reqId': 1507743286863262, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743286, 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba'}}} 2017-10-11 17:39:56,854 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': [], 'op': 'CATCHUP_REP', 'ledgerId': 1, 'txns': {'16': {'role': '2', 'reqId': 1507743286863262, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743286, 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba'}}}, 'Node4') 2017-10-11 17:39:56,854 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': [], 'ledgerId': 1, 'txns': {'16': {'role': '2', 'reqId': 1507743286863262, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743286, 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba'}}} 2017-10-11 17:39:56,854 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node2: CATCHUP_REP{'consProof': ['FPVihNkFRp2M9DiCynmHpGY9i1DfTgvDB2fSmBPrTVzG', '7s2bfj7LTYYEmRdcCimvjsTHQBpZ9bo7PeCypXFuUHf2', '4jhsGUEJzC6HEwZJEUY5FqkTC8PSeRzPqw5d6khvq7x4', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'12': {'reqId': 1507741500981867, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741501, 'dest': '542MVr22zcHbVyGzaXmbT3', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU'}, '13': {'reqId': 1507741503995435, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741504, 'dest': '542MVr22zcHbVyGzaXmbT4', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ'}}} 2017-10-11 17:39:56,854 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 2 transactions in the catchup from Node2 2017-10-11 17:39:56,854 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,854 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 2 of them now, from 12 to 13 2017-10-11 17:39:56,859 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 0 catchup replies with sequence numbers [] 2017-10-11 17:39:56,859 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node4 2017-10-11 17:39:56,859 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,860 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node4 2017-10-11 17:39:56,860 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,860 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node4: CATCHUP_REP{'consProof': [], 'ledgerId': 1, 'txns': {'16': {'role': '2', 'reqId': 1507743286863262, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743286, 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3tkLi59vF9FteMEUe2g6PHB1LLa6SfZxY9t85ENZEBX7Jvj91Yy6e7YpjUoqWR4CQGP47nte2hQSAubfjRCbntba'}}} 2017-10-11 17:39:56,860 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 1 transactions in the catchup from Node4 2017-10-11 17:39:56,860 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,861 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 3 of them now, from 12 to 16 2017-10-11 17:39:56,861 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 0 catchup replies with sequence numbers [] 2017-10-11 17:39:56,861 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,865 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CATCHUP_REP{'consProof': ['fpwT2fUkSXzf2Z2GbSxzPXpnBJxk7t6dmwaW6uSVeRV', 'BFfQcb3gKDHFRXCaFbBs57vGxSdJuRcPowbnJfFCBH7t', '9VMfXRPpWTW9htbmEuYh3vtCacg6XMGGQBasdUqHFxbA', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'10': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741252, 'dest': '542MVr22zcHbVyGzaXmbT1', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh'}, '11': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741256, 'dest': '542MVr22zcHbVyGzaXmbT2', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3'}}} 2017-10-11 17:39:56,865 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['fpwT2fUkSXzf2Z2GbSxzPXpnBJxk7t6dmwaW6uSVeRV', 'BFfQcb3gKDHFRXCaFbBs57vGxSdJuRcPowbnJfFCBH7t', '9VMfXRPpWTW9htbmEuYh3vtCacg6XMGGQBasdUqHFxbA', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'op': 'CATCHUP_REP', 'txns': {'10': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741252, 'dest': '542MVr22zcHbVyGzaXmbT1', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh'}, '11': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741256, 'dest': '542MVr22zcHbVyGzaXmbT2', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3'}}}, 'Node1') 2017-10-11 17:39:56,865 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['fpwT2fUkSXzf2Z2GbSxzPXpnBJxk7t6dmwaW6uSVeRV', 'BFfQcb3gKDHFRXCaFbBs57vGxSdJuRcPowbnJfFCBH7t', '9VMfXRPpWTW9htbmEuYh3vtCacg6XMGGQBasdUqHFxbA', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'10': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741252, 'dest': '542MVr22zcHbVyGzaXmbT1', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh'}, '11': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741256, 'dest': '542MVr22zcHbVyGzaXmbT2', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3'}}} 2017-10-11 17:39:56,865 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node1: CATCHUP_REP{'consProof': ['fpwT2fUkSXzf2Z2GbSxzPXpnBJxk7t6dmwaW6uSVeRV', 'BFfQcb3gKDHFRXCaFbBs57vGxSdJuRcPowbnJfFCBH7t', '9VMfXRPpWTW9htbmEuYh3vtCacg6XMGGQBasdUqHFxbA', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'10': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741252, 'dest': '542MVr22zcHbVyGzaXmbT1', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh'}, '11': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507741256, 'dest': '542MVr22zcHbVyGzaXmbT2', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3'}}} 2017-10-11 17:39:56,866 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 2 transactions in the catchup from Node1 2017-10-11 17:39:56,866 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,866 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 5 of them now, from 10 to 16 2017-10-11 17:39:56,866 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 11, 16, b'\x00\x18\xc2\t\xcd\x8c\x92#h;\xe2\xffiDh\xbfi\xe6\x9b\xb8\x8a\xa4\xc8i\xe2\xd5\t\xecn\x84\xfa\xc5', b'\xfa7\x90\x04\x0c\x9e\\\x8d\xb6\xc7\x9d\xbc\xd0\x12D`\xa9\x9d6E\xbe0\x9d1\x0c\xef\xe2\xf8\xa2\xcc\xe5\xc9', [b'\t\xf2Fy6F\\\xa5\xbe\xfc9\x11:!d``,nH\xe6\xf5\xf0A$(5\x0e\x80\xc9\x9f\xf0', b'\x98Vm{\xe0\xc6\xd8>>\xa9>\xf3\x94\xdf\x9f2o\xb65\xbeY\x03\x0c\xaa\xd3IL\xf3B\xe9\x80\xcf', b'~!\x1f\x87\xeb\xfc\x08ck\x99\x0c\xf2\xb1\xe2\x9eEXIt\x19\xd4\xcd\x83X\t\x94\xd6Z*\xac\xe6i', b'\x03\x7f\x0c\xef\x8aSkd\xbb+\x1e\x07T\xea9\xc9y*\x04\x95\x97\xa3\xf1X\xf2\xad_W\x00Btm', b'\xd7\r\x82\x86t\xa38`\xaa\xbb0\xa9J\xccM\xd1\xbci\xb8_~^\x18\x1a\x92\x934\xcd\xcaB\x1f\xb9'] 2017-10-11 17:39:56,870 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 13, 16, b'\x94\xf7\xaf@\xd1P\xe0\x9b\x0b*\xbe\xff\xc6q\xc2z\xbd\xa8\xee\x00\x8d\xce\x86\xc4#\xe0\x04\xebE\xa2\xb1]', b'\xfa7\x90\x04\x0c\x9e\\\x8d\xb6\xc7\x9d\xbc\xd0\x12D`\xa9\x9d6E\xbe0\x9d1\x0c\xef\xe2\xf8\xa2\xcc\xe5\xc9', [b'\xd5\xc6\xde\tc\x9f\xb0W\xe1\x8d>\xd0\x0co \x8e\x1a\x8c\xaa\x8f\x8bf2/\xb6\x03\xd5(\xa2\x9e\x16I', b'e\xf6\xf7\xf3o:`,\xe3\xe8\xec\x13\xa9|\xb9\xd5j\xac\xd1K\xfbd \xd0.\xa7\x1f\xcf3MV\x15', b'7\x83\xa9\xb0o\xe4\x83\xe1y)P\x03n\x89t\xe8\xb9\xa6i\x9c\xdb)\x11o\xd1\x0e\x11U`+]a', b'c\xcd\xa7H\xe08\x1d\xe3\x05k\xbf$\x8a~\xb2\xbc\x8e`\xde\xae\xba\xa1\x1e!\x9ah\x8b\xd5\xc2B\x97\xd5', b'\xd7\r\x82\x86t\xa38`\xaa\xbb0\xa9J\xccM\xd1\xbci\xb8_~^\x18\x1a\x92\x934\xcd\xcaB\x1f\xb9'] 2017-10-11 17:39:56,874 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 4 catchup replies with sequence numbers [10, 11, 12, 13] 2017-10-11 17:39:56,876 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:39:56,876 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"consProof":["5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G","93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT","BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA","7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"txns":{"14":{"type":"1","signature":"TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT5","txnTime":1507743135,"reqId":1507743135302714},"15":{"type":"1","signature":"5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT6","txnTime":1507743137,"reqId":1507743137917161}},"ledgerId":1,"op":"CATCHUP_REP"}'], 'signature': None} 2017-10-11 17:39:56,877 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"consProof":["5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G","93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT","BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA","7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"txns":{"14":{"type":"1","signature":"TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT5","txnTime":1507743135,"reqId":1507743135302714},"15":{"type":"1","signature":"5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT6","txnTime":1507743137,"reqId":1507743137917161}},"ledgerId":1,"op":"CATCHUP_REP"}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,877 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"ledgerId":1,"newMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","hashes":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o","JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf","Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":5,"oldMerkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","seqNoEnd":16,"seqNoStart":9}', '{"consProof":["5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G","93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT","BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA","7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"txns":{"14":{"type":"1","signature":"TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT5","txnTime":1507743135,"reqId":1507743135302714},"15":{"type":"1","signature":"5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"542MVr22zcHbVyGzaXmbT6","txnTime":1507743137,"reqId":1507743137917161}},"ledgerId":1,"op":"CATCHUP_REP"}'], 'signature': None} 2017-10-11 17:39:56,877 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,877 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node3') 2017-10-11 17:39:56,877 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,878 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,878 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9}, 'Node3') 2017-10-11 17:39:56,878 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} 2017-10-11 17:39:56,878 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CATCHUP_REP{'consProof': ['5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G', '93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT', 'BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'14': {'reqId': 1507743135302714, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743135, 'dest': '542MVr22zcHbVyGzaXmbT5', 'signature': 'TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa'}, '15': {'reqId': 1507743137917161, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743137, 'dest': '542MVr22zcHbVyGzaXmbT6', 'signature': '5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u'}}} 2017-10-11 17:39:56,878 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': ['5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G', '93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT', 'BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'op': 'CATCHUP_REP', 'txns': {'14': {'reqId': 1507743135302714, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743135, 'dest': '542MVr22zcHbVyGzaXmbT5', 'signature': 'TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa'}, '15': {'reqId': 1507743137917161, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743137, 'dest': '542MVr22zcHbVyGzaXmbT6', 'signature': '5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u'}}}, 'Node3') 2017-10-11 17:39:56,879 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'consProof': ['5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G', '93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT', 'BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'14': {'reqId': 1507743135302714, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743135, 'dest': '542MVr22zcHbVyGzaXmbT5', 'signature': 'TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa'}, '15': {'reqId': 1507743137917161, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743137, 'dest': '542MVr22zcHbVyGzaXmbT6', 'signature': '5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u'}}} 2017-10-11 17:39:56,879 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node3 2017-10-11 17:39:56,879 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,879 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'newMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ppSeqNo': 5, 'oldMerkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'seqNoEnd': 16, 'viewNo': 4, 'hashes': ['jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb', '77FQeVNKAnoYXTZ65rYxC242hZWEYSc4ooxyCHfLNw1o', 'JDT9csWTBbyCC1k9ZdxyZUJZxpJ87b4dQSw9mP1W23Qf', 'Eeb4RsaRfPjwFMUsPbC3eZMVD6LzDhWeC5nDZLFDj9N', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'seqNoStart': 9} from Node3 2017-10-11 17:39:56,880 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:39:56,880 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node3: CATCHUP_REP{'consProof': ['5yXqxv8R35Vw37GZKb9V616EtX4X6gFUc6scYLWeLv9G', '93nwxDDATP9tUvaEC6zYhz3689vFqgsVqMNvF5BWkxdT', 'BnTgScdFLB5sdUEqgnDLGevWtBehdwAiSsKbY1SNcSKA', '7ibF6aU59hrPwbZcLMXo6a8RrMHX2xLrRxm7de2wJxUL', 'FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz'], 'ledgerId': 1, 'txns': {'14': {'reqId': 1507743135302714, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743135, 'dest': '542MVr22zcHbVyGzaXmbT5', 'signature': 'TNQzGaJimAe1CJJUCBSovKsPdznKeXuDV6yy5CpU3zjf4BL8kbGCAGm7MNnCMkgr3JUYDicfw7YmiL7hyZHUspa'}, '15': {'reqId': 1507743137917161, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'type': '1', 'txnTime': 1507743137, 'dest': '542MVr22zcHbVyGzaXmbT6', 'signature': '5Q6YwS2H3HTFPF5R5YKJ4JDDoJiUgFGkiWJCNBGHsVfjKP3mtv2aXoLd98AzipsHXHuHXuKLyzNwfPoGxVTaiA7u'}}} 2017-10-11 17:39:56,880 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 2 transactions in the catchup from Node3 2017-10-11 17:39:56,880 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:39:56,880 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 3 of them now, from 14 to 16 2017-10-11 17:39:56,881 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 15, 16, b'v$\x93\x8f\xb4\x10\xe8\xb8_\x0e\xef\xa0\xa5\xc3\xed\xcd\x96\xf0\x1e\x0c\xeav\x1c\xfb $\x95\xaebOj\xc2', b'\xfa7\x90\x04\x0c\x9e\\\x8d\xb6\xc7\x9d\xbc\xd0\x12D`\xa9\x9d6E\xbe0\x9d1\x0c\xef\xe2\xf8\xa2\xcc\xe5\xc9', [b'I\xea&\x0fKq\xb2\x81\xa1x\xa8`H\xd4\xf6u\x8c\x1f}h\x8ab\xb5\x97Z\xf6}j\xe0\xdd\x10\xfb', b"w\x94\x9c'(\xb3\x7f\x88\xc9\xb8&\xe2;\xa4\xad\xae4a\xb0\xdbG\xa1\xdb\xf0\xa8\xd6\xbco\xd5\x98\xe5\xc6", b'\xa0:,A\x95\xb8y\xe8\xbd\xe9\xdb\x9f\xf1YnmV\xfc\xd3Z\xeeW\xd440lx\xc6Y\xb5*\x89', b'c\xcd\xa7H\xe08\x1d\xe3\x05k\xbf$\x8a~\xb2\xbc\x8e`\xde\xae\xba\xa1\x1e!\x9ah\x8b\xd5\xc2B\x97\xd5', b'\xd7\r\x82\x86t\xa38`\xaa\xbb0\xa9J\xccM\xd1\xbci\xb8_~^\x18\x1a\x92\x934\xcd\xcaB\x1f\xb9'] 2017-10-11 17:39:56,883 | DEBUG | ledger_manager.py ( 595) | hasValidCatchupReplies | Node5 verifying proof for 16, 16, b'\xfa7\x90\x04\x0c\x9e\\\x8d\xb6\xc7\x9d\xbc\xd0\x12D`\xa9\x9d6E\xbe0\x9d1\x0c\xef\xe2\xf8\xa2\xcc\xe5\xc9', b'\xfa7\x90\x04\x0c\x9e\\\x8d\xb6\xc7\x9d\xbc\xd0\x12D`\xa9\x9d6E\xbe0\x9d1\x0c\xef\xe2\xf8\xa2\xcc\xe5\xc9', [] 2017-10-11 17:39:56,888 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 3 catchup replies with sequence numbers [14, 15, 16] 2017-10-11 17:39:56,888 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 1, caught up 7 in total 2017-10-11 17:39:56,888 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node5 caught up to 19 txns in the last catchup 2017-10-11 17:39:56,888 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node5:0 set last ordered as (4, 6) 2017-10-11 17:39:56,888 | DEBUG | replica.py (2234) | _remove_till_caught_up_3pc | Node5:0 going to remove messages for 0 3PC keys 2017-10-11 17:39:56,888 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,888 | INFO | node.py (1525) | allLedgersCaughtUp | CATCH-UP: Node5 caught up till (4, 6) 2017-10-11 17:39:56,889 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,889 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,889 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node5 needs 2 ViewChangeDone messages 2017-10-11 17:39:56,889 | DEBUG | node.py (1568) | caught_up_for_current_view | Node5 does not have view change quorum for view 4 2017-10-11 17:39:56,889 | DEBUG | node.py (1552) | is_catchup_needed | Node5 is not caught up for the current view 4 2017-10-11 17:39:56,889 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node5 caught up to 19 txns in the last catchup 2017-10-11 17:39:56,890 | INFO | node.py (1532) | allLedgersCaughtUp | Node5 needs to catchup again 2017-10-11 17:39:56,890 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,890 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,890 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,890 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,890 | DEBUG | node.py (2192) | start_catchup | Node5 reverted 0 batches before starting catch up 2017-10-11 17:39:56,890 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,891 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 0 2017-10-11 17:39:56,891 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,891 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 0 2017-10-11 17:39:56,891 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,891 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 0 2017-10-11 17:39:56,891 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,892 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 0 2017-10-11 17:39:56,892 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:39:56,892 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 0 2017-10-11 17:39:56,892 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node2 into one transmission 2017-10-11 17:39:56,892 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-10-11 17:39:56,893 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-10-11 17:39:56,893 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node2 2017-10-11 17:39:56,893 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,893 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:39:56,893 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-10-11 17:39:56,893 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-10-11 17:39:56,893 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-10-11 17:39:56,894 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,894 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node4 into one transmission 2017-10-11 17:39:56,894 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-10-11 17:39:56,894 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-10-11 17:39:56,894 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node4 2017-10-11 17:39:56,894 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,895 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node1 into one transmission 2017-10-11 17:39:56,895 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-10-11 17:39:56,895 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-10-11 17:39:56,895 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-10-11 17:39:56,895 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,910 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 4 messages through listener 2017-10-11 17:39:56,911 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}'], 'signature': None} 2017-10-11 17:39:56,911 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,911 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}'], 'signature': None} 2017-10-11 17:39:56,911 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,912 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:39:56,912 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,912 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,912 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:39:56,912 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,912 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,912 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,913 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,913 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,913 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:39:56,913 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,913 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,913 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:39:56,914 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,914 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,914 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,914 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,914 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,914 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node2') 2017-10-11 17:39:56,915 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,915 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,915 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node2') 2017-10-11 17:39:56,915 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,915 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,915 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:56,915 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":6,"txnSeqNo":5,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,916 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,916 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node1') 2017-10-11 17:39:56,916 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,916 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,916 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}}, 'Node1') 2017-10-11 17:39:56,917 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'params': {'ledgerId': 0}} 2017-10-11 17:39:56,917 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node4 2017-10-11 17:39:56,917 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,917 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,917 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node4 2017-10-11 17:39:56,918 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,919 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,919 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node3 2017-10-11 17:39:56,919 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,919 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,920 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node3 2017-10-11 17:39:56,920 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,920 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,920 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node2 2017-10-11 17:39:56,921 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,921 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,921 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node3', 'Node4', 'Node2'} that its ledger of type 0 is latest 2017-10-11 17:39:56,921 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node5 found from ledger status LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} that it does not need catchup 2017-10-11 17:39:56,921 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:56,921 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,921 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,922 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,922 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,922 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,922 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,922 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 0 2017-10-11 17:39:56,922 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,922 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 1 and nodes set() 2017-10-11 17:39:56,924 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,924 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 2 2017-10-11 17:39:56,924 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,924 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 2 2017-10-11 17:39:56,925 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,925 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 2 2017-10-11 17:39:56,925 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,925 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 2 2017-10-11 17:39:56,925 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:39:56,925 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 2 2017-10-11 17:39:56,925 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 0 stashed ledger statuses for ledger 2 2017-10-11 17:39:56,926 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 0, caught up 0 in total 2017-10-11 17:39:56,926 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node2 2017-10-11 17:39:56,928 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,929 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,929 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node1 2017-10-11 17:39:56,930 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,930 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,930 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} from Node1 2017-10-11 17:39:56,930 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,931 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:39:56,931 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node2 into one transmission 2017-10-11 17:39:56,931 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-10-11 17:39:56,931 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-10-11 17:39:56,932 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node2 2017-10-11 17:39:56,932 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,932 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:39:56,932 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-10-11 17:39:56,932 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-10-11 17:39:56,933 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-10-11 17:39:56,933 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,933 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node4 into one transmission 2017-10-11 17:39:56,934 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-10-11 17:39:56,935 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-10-11 17:39:56,935 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-10-11 17:39:56,935 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,936 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node1 into one transmission 2017-10-11 17:39:56,936 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-10-11 17:39:56,936 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-10-11 17:39:56,936 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-10-11 17:39:56,936 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,954 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 4 messages through listener 2017-10-11 17:39:56,955 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,956 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None}, 'Node3') 2017-10-11 17:39:56,957 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:56,957 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,957 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:39:56,957 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,957 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,957 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:39:56,958 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,958 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}'], 'signature': None} 2017-10-11 17:39:56,958 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}'], 'signature': None}, 'Node4') 2017-10-11 17:39:56,958 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}'], 'signature': None} 2017-10-11 17:39:56,958 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,962 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:39:56,962 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,962 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,962 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:39:56,965 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,965 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,966 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:56,966 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,966 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,966 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node2') 2017-10-11 17:39:56,966 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,966 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,967 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node2') 2017-10-11 17:39:56,967 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,967 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,967 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:56,968 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:56,968 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,968 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:39:56,969 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,969 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,969 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:39:56,969 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'params': {'ledgerId': 2}} 2017-10-11 17:39:56,969 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,970 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,970 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,970 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node3 2017-10-11 17:39:56,972 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,973 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,973 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,973 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,973 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,974 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node4 2017-10-11 17:39:56,975 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,977 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,977 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,977 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,978 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,978 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node3', 'Node4', 'Node2'} that its ledger of type 2 is latest 2017-10-11 17:39:56,978 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node5 found from ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} that it does not need catchup 2017-10-11 17:39:56,978 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:56,978 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,978 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:56,978 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:56,978 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:56,978 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:56,979 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:56,979 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 2 2017-10-11 17:39:56,979 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,979 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-11 17:39:56,979 | DEBUG | upgrader.py ( 208) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc processing config ledger for any upgrades 2017-10-11 17:39:56,980 | INFO | upgrader.py ( 214) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,980 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node5' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,980 | INFO | upgrader.py ( 304) | handleUpgradeTxn | Node 'Node5' disregards upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:39:56,980 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:39:56,980 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 1 2017-10-11 17:39:56,981 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-11 17:39:56,981 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 1 2017-10-11 17:39:56,981 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:39:56,981 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 1 2017-10-11 17:39:56,981 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:39:56,981 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 1 2017-10-11 17:39:56,981 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:39:56,982 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 1 2017-10-11 17:39:56,982 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 0 stashed ledger statuses for ledger 1 2017-10-11 17:39:56,982 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node5' last upgrade txn is None 2017-10-11 17:39:56,982 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 2, caught up 0 in total 2017-10-11 17:39:56,982 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node2 2017-10-11 17:39:56,983 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,983 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,984 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,985 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,985 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,985 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} from Node1 2017-10-11 17:39:56,987 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,988 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:39:56,989 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node2 into one transmission 2017-10-11 17:39:56,989 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-10-11 17:39:56,989 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-10-11 17:39:56,989 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node2 2017-10-11 17:39:56,989 | DEBUG | zstack.py ( 728) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,990 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:39:56,990 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-10-11 17:39:56,990 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-10-11 17:39:56,990 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-10-11 17:39:56,990 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,990 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node4 into one transmission 2017-10-11 17:39:56,990 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-10-11 17:39:56,991 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-10-11 17:39:56,991 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-10-11 17:39:56,991 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:56,991 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node1 into one transmission 2017-10-11 17:39:56,991 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-10-11 17:39:56,991 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-10-11 17:39:56,991 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-10-11 17:39:56,992 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:39:57,016 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 4 messages through listener 2017-10-11 17:39:57,018 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:57,019 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node2') 2017-10-11 17:39:57,020 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:57,020 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,021 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node2') 2017-10-11 17:39:57,021 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,022 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,022 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node2') 2017-10-11 17:39:57,022 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,023 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:57,023 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node1') 2017-10-11 17:39:57,023 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:39:57,024 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,024 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:39:57,025 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,025 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,026 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:39:57,026 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,029 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:57,029 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None}, 'Node3') 2017-10-11 17:39:57,030 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:39:57,030 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,031 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:39:57,031 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,031 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,032 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:39:57,032 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,032 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None} 2017-10-11 17:39:57,033 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None}, 'Node4') 2017-10-11 17:39:57,033 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":5,"txnSeqNo":16,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None} 2017-10-11 17:39:57,034 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,034 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:39:57,034 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,035 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,035 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:39:57,035 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'}, 'params': {'ledgerId': 1}} 2017-10-11 17:39:57,036 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:57,036 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,037 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,037 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node2 2017-10-11 17:39:57,038 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,038 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,038 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:57,040 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,041 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,041 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node1 2017-10-11 17:39:57,041 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,042 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,042 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:57,042 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,043 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,043 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node3', 'Node2', 'Node1'} that its ledger of type 1 is latest 2017-10-11 17:39:57,043 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node5 found from ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} that it does not need catchup 2017-10-11 17:39:57,043 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:39:57,043 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:57,044 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:39:57,044 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:39:57,044 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:39:57,046 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:57,046 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:57,046 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 1 2017-10-11 17:39:57,047 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,047 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 1, caught up 0 in total 2017-10-11 17:39:57,047 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node5 caught up to 0 txns in the last catchup 2017-10-11 17:39:57,047 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:39:57,047 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:39:57,047 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node5 needs 2 ViewChangeDone messages 2017-10-11 17:39:57,048 | DEBUG | node.py (1568) | caught_up_for_current_view | Node5 does not have view change quorum for view 4 2017-10-11 17:39:57,048 | DEBUG | node.py (1552) | is_catchup_needed | Node5 is not caught up for the current view 4 2017-10-11 17:39:57,048 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node5 caught up to 0 txns in the last catchup 2017-10-11 17:39:57,048 | DEBUG | node.py (1557) | is_catchup_needed | Node5 ordered till last prepared certificate 2017-10-11 17:39:57,048 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node5 does not need any more catchups 2017-10-11 17:39:57,049 | DEBUG | primary_decider.py ( 134) | send | Node5's elector sending VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4} 2017-10-11 17:39:57,049 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node5 needs 1 ViewChangeDone messages 2017-10-11 17:39:57,049 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:39:57,050 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node3 2017-10-11 17:39:57,050 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,051 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,051 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:57,052 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,052 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,052 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': 4, 'ppSeqNo': 5, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} from Node4 2017-10-11 17:39:57,052 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:39:57,053 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 16 2017-10-11 17:40:01,777 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_txns_if_needed) with id 2 2017-10-11 17:40:01,777 | DEBUG | ledger_manager.py ( 136) | request_txns_if_needed | Node5 not missing any transactions for ledger 0 2017-10-11 17:40:06,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:06,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:06,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:06,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:06,320 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:06,320 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:06,320 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 83 43264272 2017-10-11 17:40:06,320 | TRACE | remote.py ( 86) | connect | connecting socket 92 43205952 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:06,320 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:40:06,320 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:06,320 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:06,321 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 84 43139728 2017-10-11 17:40:06,321 | TRACE | remote.py ( 86) | connect | connecting socket 79 43264208 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:06,321 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:40:06,321 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:06,321 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:06,321 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 86 43319616 2017-10-11 17:40:06,322 | TRACE | remote.py ( 86) | connect | connecting socket 84 43139728 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:06,322 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:40:06,325 | DEBUG | zstack.py ( 605) | reconnectRemote | Node5 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:06,325 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:06,325 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 91 43247424 2017-10-11 17:40:06,325 | TRACE | remote.py ( 86) | connect | connecting socket 86 43319616 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:06,326 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:40:06,326 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:40:06,338 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:40:06,340 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:40:06,340 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:40:06,340 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:40:06,341 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:40:06,341 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:40:06,341 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:40:06,341 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:40:06,352 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:40:06,353 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node4 2017-10-11 17:40:06,355 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from set() to {'Node4'} 2017-10-11 17:40:06,355 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node4 2017-10-11 17:40:06,356 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} to 1 recipients: ['Node4'] 2017-10-11 17:40:06,358 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} to 1 recipients: ['Node4'] 2017-10-11 17:40:06,358 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} to 1 recipients: ['Node4'] 2017-10-11 17:40:06,359 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node4 into one transmission 2017-10-11 17:40:06,359 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":5,"viewNo":null,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":16,"viewNo":null,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":11,"viewNo":null,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}']) 2017-10-11 17:40:06,359 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}"],"signature":null}' 2017-10-11 17:40:06,360 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}"],"signature":null}' to Node4 2017-10-11 17:40:06,371 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:40:06,372 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node3 2017-10-11 17:40:06,372 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node1 2017-10-11 17:40:06,372 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node2 2017-10-11 17:40:06,373 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from {'Node4'} to {'Node3', 'Node2', 'Node4', 'Node1'} 2017-10-11 17:40:06,373 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node3 2017-10-11 17:40:06,373 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node1 2017-10-11 17:40:06,373 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node2 2017-10-11 17:40:06,373 | DEBUG | motor.py ( 34) | set_status | Node5 changing status from starting to started 2017-10-11 17:40:06,374 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 5 and nodes {'Node3', 'Node2', 'Node4', 'Node1'} 2017-10-11 17:40:06,375 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to lagged node Node3 2017-10-11 17:40:06,376 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to 1 recipients: ['Node3'] 2017-10-11 17:40:06,376 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to lagged node Node1 2017-10-11 17:40:06,376 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to 1 recipients: ['Node1'] 2017-10-11 17:40:06,377 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to lagged node Node2 2017-10-11 17:40:06,377 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4}], 'viewNo': 4} to 1 recipients: ['Node2'] 2017-10-11 17:40:06,377 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} to 1 recipients: ['Node3'] 2017-10-11 17:40:06,377 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} to 1 recipients: ['Node3'] 2017-10-11 17:40:06,378 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} to 1 recipients: ['Node3'] 2017-10-11 17:40:06,379 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} to 1 recipients: ['Node1'] 2017-10-11 17:40:06,379 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} to 1 recipients: ['Node1'] 2017-10-11 17:40:06,379 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} to 1 recipients: ['Node1'] 2017-10-11 17:40:06,380 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'} to 1 recipients: ['Node2'] 2017-10-11 17:40:06,380 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 16, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'} to 1 recipients: ['Node2'] 2017-10-11 17:40:06,380 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'} to 1 recipients: ['Node2'] 2017-10-11 17:40:06,381 | DEBUG | node.py (2608) | send | Node5 sending message VIEW_CHANGE_DONE{'name': 'Node5', 'ledgerInfo': [(0, 5, '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'), (1, 16, 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'), (2, 11, 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F')], 'viewNo': 4} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:40:06,381 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:40:06,381 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node5 got view change quorum (2 >= 2) 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node5 received ViewChangeDone from primary Node5 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 244) | has_sufficient_same_view_change_done_messages | Node5 does not have acceptable primary, only 1 votes for ('Node1', ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'), (2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN'))) 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node5 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'), (2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN')) 2017-10-11 17:40:06,382 | ERROR | primary_selector.py ( 167) | _verify_primary | PRIMARY SELECTION: Node5 expected next primary to be Node5, but majority declared Node1 instead for view 4 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:40:06,382 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:40:06,383 | ERROR | primary_selector.py ( 167) | _verify_primary | PRIMARY SELECTION: Node5 expected next primary to be Node5, but majority declared Node1 instead for view 4 2017-10-11 17:40:06,383 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:40:06,383 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']], 'viewNo': 4} 2017-10-11 17:40:06,383 | ERROR | primary_selector.py ( 167) | _verify_primary | PRIMARY SELECTION: Node5 expected next primary to be Node5, but majority declared Node1 instead for view 4 2017-10-11 17:40:06,383 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:40:06,384 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node2 into one transmission 2017-10-11 17:40:06,384 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"name":"Node5","viewNo":4}],"viewNo":4}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":5,"viewNo":null,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":16,"viewNo":null,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":11,"viewNo":null,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', b'{"name":"Node5","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"viewNo":4}']) 2017-10-11 17:40:06,384 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' 2017-10-11 17:40:06,384 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' to Node2 2017-10-11 17:40:06,384 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node3 into one transmission 2017-10-11 17:40:06,385 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"name":"Node5","viewNo":4}],"viewNo":4}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":5,"viewNo":null,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":16,"viewNo":null,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":11,"viewNo":null,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', b'{"name":"Node5","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"viewNo":4}']) 2017-10-11 17:40:06,385 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' 2017-10-11 17:40:06,385 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' to Node3 2017-10-11 17:40:06,385 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"name":"Node5","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"viewNo":4}' to Node4 2017-10-11 17:40:06,385 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"name":"Node5","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"viewNo":4}' to Node4 2017-10-11 17:40:06,388 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node1 into one transmission 2017-10-11 17:40:06,388 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"name":"Node5","viewNo":4}],"viewNo":4}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":0,"txnSeqNo":5,"viewNo":null,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":1,"txnSeqNo":16,"viewNo":null,"merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', b'{"ppSeqNo":null,"op":"LEDGER_STATUS","ledgerId":2,"txnSeqNo":11,"viewNo":null,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"}', b'{"name":"Node5","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,5,"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7"],[1,16,"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"],[2,11,"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F"]],"viewNo":4}']) 2017-10-11 17:40:06,388 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' 2017-10-11 17:40:06,388 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"name\\":\\"Node5\\",\\"viewNo\\":4}],\\"viewNo\\":4}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"txnSeqNo\\":5,\\"viewNo\\":null,\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":1,\\"txnSeqNo\\":16,\\"viewNo\\":null,\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"}","{\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":2,\\"txnSeqNo\\":11,\\"viewNo\\":null,\\"merkleRoot\\":\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"}","{\\"name\\":\\"Node5\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,5,\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\"],[1,16,\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\"],[2,11,\\"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F\\"]],\\"viewNo\\":4}"],"signature":null}' to Node1 2017-10-11 17:40:06,708 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 184 2017-10-11 17:40:06,709 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:06,709 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 185 to run in 10 seconds 2017-10-11 17:40:16,667 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_CPs_if_needed) with id 1 2017-10-11 17:40:16,715 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 185 2017-10-11 17:40:16,715 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:16,716 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 186 to run in 10 seconds 2017-10-11 17:40:16,752 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_CPs_if_needed) with id 3 2017-10-11 17:40:16,839 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_CPs_if_needed) with id 5 2017-10-11 17:40:17,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_txns_if_needed) with id 6 2017-10-11 17:40:17,638 | DEBUG | ledger_manager.py ( 136) | request_txns_if_needed | Node5 not missing any transactions for ledger 1 2017-10-11 17:40:17,948 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action partial(request_txns_if_needed) with id 4 2017-10-11 17:40:17,948 | DEBUG | ledger_manager.py ( 136) | request_txns_if_needed | Node5 not missing any transactions for ledger 2 2017-10-11 17:40:21,321 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:21,322 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:21,322 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:21,322 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:21,322 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:40:26,720 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 186 2017-10-11 17:40:26,720 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:26,720 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 187 to run in 10 seconds 2017-10-11 17:40:35,919 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 179 2017-10-11 17:40:35,919 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:40:35,920 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 188 to run in 60 seconds 2017-10-11 17:40:35,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 23 2017-10-11 17:40:35,920 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:40:35,920 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 24 to run in 60 seconds 2017-10-11 17:40:35,955 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 180 2017-10-11 17:40:35,956 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 189 to run in 60 seconds 2017-10-11 17:40:36,328 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:36,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:36,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:36,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:36,329 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:40:36,731 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 187 2017-10-11 17:40:36,732 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:36,732 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 190 to run in 10 seconds 2017-10-11 17:40:46,743 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 190 2017-10-11 17:40:46,743 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:46,743 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 191 to run in 10 seconds 2017-10-11 17:40:51,335 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:40:51,336 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:40:51,336 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:40:51,336 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:40:51,337 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:40:56,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action _check_view_change_completed with id 183 2017-10-11 17:40:56,596 | DEBUG | node.py (1011) | _check_view_change_completed | Node5 running the scheduled check for view change completion 2017-10-11 17:40:56,596 | INFO | node.py (1019) | _check_view_change_completed | view change to view 4 is not completed in time, starting view change for view 5 2017-10-11 17:40:56,597 | INFO | node.py (1021) | _check_view_change_completed | VIEW CHANGE: Node5 initiating a view change to 5 from 4 2017-10-11 17:40:56,597 | DEBUG | throttler.py ( 31) | acquire | now: 2272742.054715409, len(actionsLog): 0 2017-10-11 17:40:56,597 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-10-11 17:40:56,598 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-10-11 17:40:56,598 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node5 sending an instance change with view_no 5 since View change could not complete in time 2017-10-11 17:40:56,598 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node5 metrics for monitor: Node5 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [2271341.219211642, 2271341.220000871] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-10-11 17:40:56,599 | DEBUG | node.py (2608) | send | Node5 sending message INSTANCE_CHANGE{'reason': 28, 'viewNo': 5} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:40:56,600 | DEBUG | node.py (1958) | do_view_change_if_possible | Node5 has no quorum for view 5 2017-10-11 17:40:56,600 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node2 2017-10-11 17:40:56,601 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node2 2017-10-11 17:40:56,601 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node3 2017-10-11 17:40:56,601 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node3 2017-10-11 17:40:56,602 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node4 2017-10-11 17:40:56,602 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node4 2017-10-11 17:40:56,602 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node1 2017-10-11 17:40:56,603 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"INSTANCE_CHANGE","reason":28,"viewNo":5}' to Node1 2017-10-11 17:40:56,754 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 191 2017-10-11 17:40:56,754 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:40:56,754 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 192 to run in 10 seconds 2017-10-11 17:41:06,344 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:41:06,345 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:41:06,345 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:41:06,345 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:41:06,345 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:41:06,765 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 192 2017-10-11 17:41:06,765 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:06,765 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 193 to run in 10 seconds 2017-10-11 17:41:16,769 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 193 2017-10-11 17:41:16,769 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:16,769 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 194 to run in 10 seconds 2017-10-11 17:41:21,353 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:41:21,353 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:41:21,353 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:41:21,353 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:41:21,354 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:41:26,775 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 194 2017-10-11 17:41:26,775 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:26,775 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 195 to run in 10 seconds 2017-10-11 17:41:33,931 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:41:33,933 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:41:33,934 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,934 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,938 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:41:33,946 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,948 | INFO | propagator.py ( 148) | propagate | Node5 propagating request ('XhYtvJqezMUKfF6KVNaGmT', 1507743693883477) from client b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:41:33,948 | DEBUG | node.py (2608) | send | Node5 sending message PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:41:33,949 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since not finalised 2017-10-11 17:41:33,950 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743693883477,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:41:33,951 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node2 2017-10-11 17:41:33,951 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node2 2017-10-11 17:41:33,951 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node3 2017-10-11 17:41:33,952 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node3 2017-10-11 17:41:33,952 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node4 2017-10-11 17:41:33,952 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node4 2017-10-11 17:41:33,953 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node1 2017-10-11 17:41:33,953 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743693883477,"operation":{"dest":"542MVr22zcHbVyGzaXmbT7","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node1 2017-10-11 17:41:33,954 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:33,957 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743693883477 2017-10-11 17:41:33,957 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,957 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node3') 2017-10-11 17:41:33,957 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,958 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743693883477 2017-10-11 17:41:33,960 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,961 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node2') 2017-10-11 17:41:33,961 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,961 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,961 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,962 | DEBUG | propagator.py ( 214) | forward | Node5 forwarding request ('XhYtvJqezMUKfF6KVNaGmT', 1507743693883477) to 0 replicas 2017-10-11 17:41:33,962 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,962 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,962 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:41:33,977 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:33,978 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743693883477 2017-10-11 17:41:33,978 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,979 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node1') 2017-10-11 17:41:33,979 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,979 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': None, 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743693, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:33,980 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'discarded': 1, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'ledgerId': 1, 'op': 'PREPREPARE', 'instId': 1, 'viewNo': 4}, 'Node2') 2017-10-11 17:41:33,980 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': None, 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743693, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:33,980 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,980 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,980 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:41:33,983 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': None, 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743693, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 1, 'viewNo': 4} from Node2 2017-10-11 17:41:33,984 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:33,985 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743693883477 2017-10-11 17:41:33,985 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,985 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node4') 2017-10-11 17:41:33,985 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,986 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:33,986 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'op': 'PREPARE', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4}, 'Node3') 2017-10-11 17:41:33,986 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:33,986 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:33,995 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:33,996 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:41:33,997 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} from Node3 2017-10-11 17:41:34,015 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:34,016 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:34,017 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'op': 'PREPARE', 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4}, 'Node4') 2017-10-11 17:41:34,017 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:34,019 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} from Node4 2017-10-11 17:41:34,057 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:41:34,058 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","reqIdr":[["XhYtvJqezMUKfF6KVNaGmT",1507743693883477]],"ppTime":1507743694,"discarded":1,"instId":0,"ledgerId":1,"ppSeqNo":7,"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPREPARE"}', '{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","ppTime":1507743693,"instId":1,"ppSeqNo":7,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'signature': None} 2017-10-11 17:41:34,059 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","reqIdr":[["XhYtvJqezMUKfF6KVNaGmT",1507743693883477]],"ppTime":1507743694,"discarded":1,"instId":0,"ledgerId":1,"ppSeqNo":7,"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPREPARE"}', '{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","ppTime":1507743693,"instId":1,"ppSeqNo":7,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'signature': None}, 'Node1') 2017-10-11 17:41:34,059 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","reqIdr":[["XhYtvJqezMUKfF6KVNaGmT",1507743693883477]],"ppTime":1507743694,"discarded":1,"instId":0,"ledgerId":1,"ppSeqNo":7,"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPREPARE"}', '{"viewNo":4,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","ppTime":1507743693,"instId":1,"ppSeqNo":7,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'signature': None} 2017-10-11 17:41:34,059 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PREPREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743694, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,059 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'ppTime': 1507743694, 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'discarded': 1, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 0, 'viewNo': 4, 'op': 'PREPREPARE'}, 'Node1') 2017-10-11 17:41:34,060 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743694, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,060 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:34,060 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'op': 'PREPARE', 'instId': 1, 'viewNo': 4}, 'Node1') 2017-10-11 17:41:34,060 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:34,060 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","ppSeqNo":7,"viewNo":4,"op":"PREPARE","ppTime":1507743694,"instId":0,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97"}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":7}'], 'signature': None} 2017-10-11 17:41:34,061 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","ppSeqNo":7,"viewNo":4,"op":"PREPARE","ppTime":1507743694,"instId":0,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97"}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":7}'], 'signature': None}, 'Node3') 2017-10-11 17:41:34,061 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","ppSeqNo":7,"viewNo":4,"op":"PREPARE","ppTime":1507743694,"instId":0,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97"}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":7}'], 'signature': None} 2017-10-11 17:41:34,061 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,061 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'op': 'PREPARE', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4}, 'Node3') 2017-10-11 17:41:34,061 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,062 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,062 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 7}, 'Node3') 2017-10-11 17:41:34,062 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,062 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,062 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 7}, 'Node1') 2017-10-11 17:41:34,063 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,065 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743693883477]], 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'discarded': 1, 'ppTime': 1507743694, 'ledgerId': 1, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'instId': 0, 'viewNo': 4} from Node1 2017-10-11 17:41:34,065 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} from Node3 2017-10-11 17:41:34,065 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743693, 'instId': 1, 'viewNo': 4} from Node1 2017-10-11 17:41:34,065 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} from Node3 2017-10-11 17:41:34,065 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} from Node1 2017-10-11 17:41:34,066 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:34,066 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'messages': ['{"ppTime":1507743694,"instId":0,"viewNo":4,"ppSeqNo":7,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE"}', '{"viewNo":4,"ppSeqNo":7,"op":"COMMIT","instId":1}'], 'signature': None} 2017-10-11 17:41:34,067 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"ppTime":1507743694,"instId":0,"viewNo":4,"ppSeqNo":7,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE"}', '{"viewNo":4,"ppSeqNo":7,"op":"COMMIT","instId":1}'], 'signature': None}, 'Node2') 2017-10-11 17:41:34,067 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"ppTime":1507743694,"instId":0,"viewNo":4,"ppSeqNo":7,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE"}', '{"viewNo":4,"ppSeqNo":7,"op":"COMMIT","instId":1}'], 'signature': None} 2017-10-11 17:41:34,067 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,067 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'op': 'PREPARE', 'instId': 0, 'viewNo': 4}, 'Node2') 2017-10-11 17:41:34,067 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,067 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,068 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 7}, 'Node2') 2017-10-11 17:41:34,068 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,069 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} from Node2 2017-10-11 17:41:34,069 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} from Node2 2017-10-11 17:41:34,082 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:34,082 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":4,"ppTime":1507743694,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH"}', '{"instId":1,"op":"COMMIT","viewNo":4,"ppSeqNo":7}'], 'signature': None} 2017-10-11 17:41:34,083 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":4,"ppTime":1507743694,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH"}', '{"instId":1,"op":"COMMIT","viewNo":4,"ppSeqNo":7}'], 'signature': None}, 'Node4') 2017-10-11 17:41:34,083 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"instId":0,"ppSeqNo":7,"viewNo":4,"ppTime":1507743694,"digest":"0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97","stateRootHash":"12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU","op":"PREPARE","txnRootHash":"2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH"}', '{"instId":1,"op":"COMMIT","viewNo":4,"ppSeqNo":7}'], 'signature': None} 2017-10-11 17:41:34,083 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,083 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'op': 'PREPARE', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4}, 'Node4') 2017-10-11 17:41:34,084 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:34,084 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,084 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 7}, 'Node4') 2017-10-11 17:41:34,084 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,084 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,085 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 7}, 'Node4') 2017-10-11 17:41:34,085 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,087 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': '2dUrM4EHWmmxVpG51n9viHmcgbtsCFjAzbYyRdLoyRBH', 'stateRootHash': '12h4jFGXit9HZmrpSCgWZAdjmf7ppRuKE9AhivmXgseU', 'ppSeqNo': 7, 'digest': '0d222ef0a4f69e3c08a67b9781b3bcd9fc3d889e6d6d0430b9bcdb64330d1e97', 'ppTime': 1507743694, 'instId': 0, 'viewNo': 4} from Node4 2017-10-11 17:41:34,087 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} from Node4 2017-10-11 17:41:34,087 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 7} from Node4 2017-10-11 17:41:34,088 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:34,088 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,089 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 7}, 'Node2') 2017-10-11 17:41:34,089 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,089 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} from Node2 2017-10-11 17:41:34,095 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:34,095 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,096 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 7}, 'Node3') 2017-10-11 17:41:34,096 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,097 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} from Node3 2017-10-11 17:41:34,109 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:34,110 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,110 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 7}, 'Node1') 2017-10-11 17:41:34,110 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} 2017-10-11 17:41:34,111 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 7} from Node1 2017-10-11 17:41:35,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 188 2017-10-11 17:41:35,927 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:41:35,928 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 196 to run in 60 seconds 2017-10-11 17:41:35,928 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 24 2017-10-11 17:41:35,929 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:41:35,929 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 25 to run in 60 seconds 2017-10-11 17:41:35,965 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 189 2017-10-11 17:41:35,966 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 197 to run in 60 seconds 2017-10-11 17:41:36,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:41:36,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:41:36,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:41:36,355 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:41:36,355 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:41:36,783 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 195 2017-10-11 17:41:36,783 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:36,783 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 198 to run in 10 seconds 2017-10-11 17:41:46,786 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 198 2017-10-11 17:41:46,786 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:46,786 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 199 to run in 10 seconds 2017-10-11 17:41:47,172 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:41:47,173 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743707150215 2017-10-11 17:41:47,173 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,174 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,174 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:41:47,175 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,175 | INFO | propagator.py ( 148) | propagate | Node5 propagating request ('XhYtvJqezMUKfF6KVNaGmT', 1507743707150215) from client b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:41:47,176 | DEBUG | node.py (2608) | send | Node5 sending message PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} to all recipients: ['Node2', 'Node3', 'Node4', 'Node1'] 2017-10-11 17:41:47,176 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since not finalised 2017-10-11 17:41:47,177 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743707150215,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:41:47,177 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node2 2017-10-11 17:41:47,178 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node2 2017-10-11 17:41:47,179 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node3 2017-10-11 17:41:47,183 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node3 2017-10-11 17:41:47,184 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node4 2017-10-11 17:41:47,184 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node4 2017-10-11 17:41:47,185 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node1 2017-10-11 17:41:47,186 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"op":"PROPAGATE","request":{"reqId":1507743707150215,"operation":{"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"},"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD"},"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}' to Node1 2017-10-11 17:41:47,187 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:47,193 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743707150215 2017-10-11 17:41:47,193 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,194 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node1') 2017-10-11 17:41:47,194 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,195 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,195 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,196 | DEBUG | propagator.py ( 214) | forward | Node5 forwarding request ('XhYtvJqezMUKfF6KVNaGmT', 1507743707150215) to 0 replicas 2017-10-11 17:41:47,197 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:47,199 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743707150215 2017-10-11 17:41:47,200 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,200 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node4') 2017-10-11 17:41:47,201 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,202 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,203 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,203 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:41:47,216 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:47,219 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743707150215 2017-10-11 17:41:47,219 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,219 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node2') 2017-10-11 17:41:47,220 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,220 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,221 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,221 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:41:47,254 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:47,255 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PREPREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,256 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'ppTime': 1507743707, 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'discarded': 1, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 0, 'viewNo': 4, 'op': 'PREPREPARE'}, 'Node1') 2017-10-11 17:41:47,256 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,257 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': None, 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,257 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'discarded': 1, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'ledgerId': 1, 'op': 'PREPREPARE', 'instId': 1, 'viewNo': 4}, 'Node2') 2017-10-11 17:41:47,257 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': None, 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,258 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 0, 'viewNo': 4} from Node1 2017-10-11 17:41:47,259 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPREPARE{'txnRootHash': None, 'reqIdr': [['XhYtvJqezMUKfF6KVNaGmT', 1507743707150215]], 'stateRootHash': None, 'ppSeqNo': 8, 'discarded': 1, 'ppTime': 1507743707, 'ledgerId': 1, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'instId': 1, 'viewNo': 4} from Node2 2017-10-11 17:41:47,278 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 4 messages through listener 2017-10-11 17:41:47,278 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","op":"PREPARE","txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM"}', '{"instId":1,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":null,"op":"PREPARE","txnRootHash":null}'], 'signature': None} 2017-10-11 17:41:47,279 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","op":"PREPARE","txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM"}', '{"instId":1,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":null,"op":"PREPARE","txnRootHash":null}'], 'signature': None}, 'Node4') 2017-10-11 17:41:47,280 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"instId":0,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","op":"PREPARE","txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM"}', '{"instId":1,"ppSeqNo":8,"viewNo":4,"ppTime":1507743707,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c","stateRootHash":null,"op":"PREPARE","txnRootHash":null}'], 'signature': None} 2017-10-11 17:41:47,281 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,281 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'op': 'PREPARE', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4}, 'Node4') 2017-10-11 17:41:47,282 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,282 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,283 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'op': 'PREPARE', 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4}, 'Node4') 2017-10-11 17:41:47,283 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,284 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on propagate request 1507743707150215 2017-10-11 17:41:47,285 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,285 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'}, 'Node3') 2017-10-11 17:41:47,285 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,293 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,293 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'op': 'PREPARE', 'instId': 0, 'viewNo': 4}, 'Node2') 2017-10-11 17:41:47,294 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,294 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":0,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}', '{"txnRootHash":null,"stateRootHash":null,"ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":1,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}'], 'signature': None} 2017-10-11 17:41:47,295 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":0,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}', '{"txnRootHash":null,"stateRootHash":null,"ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":1,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}'], 'signature': None}, 'Node3') 2017-10-11 17:41:47,295 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"txnRootHash":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","stateRootHash":"JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N","ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":0,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}', '{"txnRootHash":null,"stateRootHash":null,"ppSeqNo":8,"viewNo":4,"op":"PREPARE","ppTime":1507743707,"instId":1,"digest":"ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c"}'], 'signature': None} 2017-10-11 17:41:47,296 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,297 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'op': 'PREPARE', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4}, 'Node3') 2017-10-11 17:41:47,297 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} 2017-10-11 17:41:47,297 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,297 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'op': 'PREPARE', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4}, 'Node3') 2017-10-11 17:41:47,297 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,298 | DEBUG | node.py (1757) | processPropagate | Node Node5 received propagated request: PROPAGATE{'request': {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} 2017-10-11 17:41:47,298 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:41:47,298 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:41:47,299 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} from Node4 2017-10-11 17:41:47,299 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} from Node2 2017-10-11 17:41:47,299 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request PREPARE{'txnRootHash': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'stateRootHash': 'JE74BNcnihEogx47jVHzcG2ze2wY7GmKvBdPtDFDW33N', 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 0, 'viewNo': 4} from Node3 2017-10-11 17:41:47,300 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} from Node4 2017-10-11 17:41:47,300 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} from Node3 2017-10-11 17:41:47,300 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:41:47,301 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,301 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'op': 'PREPARE', 'instId': 1, 'viewNo': 4}, 'Node1') 2017-10-11 17:41:47,301 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} 2017-10-11 17:41:47,301 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'messages': ['{"instId":0,"viewNo":4,"op":"COMMIT","ppSeqNo":8}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":8}'], 'signature': None} 2017-10-11 17:41:47,302 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"viewNo":4,"op":"COMMIT","ppSeqNo":8}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":8}'], 'signature': None}, 'Node3') 2017-10-11 17:41:47,302 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"instId":0,"viewNo":4,"op":"COMMIT","ppSeqNo":8}', '{"instId":1,"viewNo":4,"op":"COMMIT","ppSeqNo":8}'], 'signature': None} 2017-10-11 17:41:47,302 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,302 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 8}, 'Node3') 2017-10-11 17:41:47,302 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,302 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,302 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 8}, 'Node3') 2017-10-11 17:41:47,303 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,303 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,303 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 8}, 'Node2') 2017-10-11 17:41:47,303 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,304 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} from Node3 2017-10-11 17:41:47,305 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} from Node2 2017-10-11 17:41:47,305 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request PREPARE{'txnRootHash': None, 'stateRootHash': None, 'ppSeqNo': 8, 'digest': 'ec304b04e040627e9843e1b307c464a849b8ea3222104fd916b8ef0966d03a0c', 'ppTime': 1507743707, 'instId': 1, 'viewNo': 4} from Node1 2017-10-11 17:41:47,305 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} from Node3 2017-10-11 17:41:47,311 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:41:47,311 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'messages': ['{"viewNo":4,"ppSeqNo":8,"instId":0,"op":"COMMIT"}', '{"viewNo":4,"ppSeqNo":8,"instId":1,"op":"COMMIT"}'], 'signature': None} 2017-10-11 17:41:47,311 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":4,"ppSeqNo":8,"instId":0,"op":"COMMIT"}', '{"viewNo":4,"ppSeqNo":8,"instId":1,"op":"COMMIT"}'], 'signature': None}, 'Node1') 2017-10-11 17:41:47,312 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'messages': ['{"viewNo":4,"ppSeqNo":8,"instId":0,"op":"COMMIT"}', '{"viewNo":4,"ppSeqNo":8,"instId":1,"op":"COMMIT"}'], 'signature': None} 2017-10-11 17:41:47,312 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,312 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 8}, 'Node1') 2017-10-11 17:41:47,312 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,312 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,313 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 8}, 'Node1') 2017-10-11 17:41:47,313 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,313 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,313 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 0, 'viewNo': 4, 'ppSeqNo': 8}, 'Node4') 2017-10-11 17:41:47,313 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,314 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} from Node1 2017-10-11 17:41:47,314 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:0 pended request COMMIT{'instId': 0, 'viewNo': 4, 'ppSeqNo': 8} from Node4 2017-10-11 17:41:47,314 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} from Node1 2017-10-11 17:41:47,315 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:47,315 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,315 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 8}, 'Node2') 2017-10-11 17:41:47,316 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,316 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} from Node2 2017-10-11 17:41:47,329 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:41:47,330 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,330 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'COMMIT', 'instId': 1, 'viewNo': 4, 'ppSeqNo': 8}, 'Node4') 2017-10-11 17:41:47,330 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} 2017-10-11 17:41:47,331 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node5:1 pended request COMMIT{'instId': 1, 'viewNo': 4, 'ppSeqNo': 8} from Node4 2017-10-11 17:41:48,913 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:41:48,914 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:41:48,914 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:48,914 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:48,914 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:41:48,915 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:41:48,915 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:41:48,915 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743693883477,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:41:51,363 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:41:51,363 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:41:51,363 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:41:51,363 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:41:51,364 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:41:56,795 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 199 2017-10-11 17:41:56,795 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:41:56,795 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 200 to run in 10 seconds 2017-10-11 17:42:03,939 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:03,940 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: BATCH{'messages': ['{"reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","identifier":"XhYtvJqezMUKfF6KVNaGmT"}', '{"reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","identifier":"XhYtvJqezMUKfF6KVNaGmT"}'], 'signature': None} 2017-10-11 17:42:03,941 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:42:03,942 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:03,943 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743707150215 2017-10-11 17:42:03,943 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:03,943 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:03,944 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:03,944 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:03,944 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:42:03,944 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743693883477,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:03,945 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:03,945 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:03,945 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:03,946 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:42:03,946 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743707150215,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:06,374 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:42:06,374 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:42:06,375 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:42:06,375 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:42:06,375 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:42:06,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 200 2017-10-11 17:42:06,800 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:42:06,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 201 to run in 10 seconds 2017-10-11 17:42:16,801 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 201 2017-10-11 17:42:16,801 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:42:16,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 202 to run in 10 seconds 2017-10-11 17:42:18,956 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:18,956 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: BATCH{'messages': ['{"reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","identifier":"XhYtvJqezMUKfF6KVNaGmT"}', '{"reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","identifier":"XhYtvJqezMUKfF6KVNaGmT"}'], 'signature': None} 2017-10-11 17:42:18,957 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:42:18,957 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:18,958 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743707150215 2017-10-11 17:42:18,958 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:18,958 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:18,959 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:18,959 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:18,959 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:42:18,959 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743693883477,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:18,959 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:18,960 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:18,960 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:18,960 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:42:18,961 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743707150215,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:21,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:42:21,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:42:21,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:42:21,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:42:21,387 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:42:26,809 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 202 2017-10-11 17:42:26,809 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:42:26,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 203 to run in 10 seconds 2017-10-11 17:42:33,972 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:33,973 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: BATCH{'messages': ['{"reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","identifier":"XhYtvJqezMUKfF6KVNaGmT"}', '{"reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","identifier":"XhYtvJqezMUKfF6KVNaGmT"}'], 'signature': None} 2017-10-11 17:42:33,974 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:42:33,974 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:33,975 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743707150215 2017-10-11 17:42:33,975 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:33,975 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:33,976 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:33,976 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} 2017-10-11 17:42:33,976 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743693883477, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT7', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3'} to its replicas since already forwarded 2017-10-11 17:42:33,976 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743693883477,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:33,976 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:33,976 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:33,977 | TRACE | propagator.py ( 141) | propagate | Node5 already propagated SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} 2017-10-11 17:42:33,977 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'reqId': 1507743707150215, 'operation': {'dest': '542MVr22zcHbVyGzaXmbT8', 'type': '1'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD'} to its replicas since already forwarded 2017-10-11 17:42:33,977 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"reqId":1507743707150215,"op":"REQACK","identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:35,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkNodeRequestSpike with id 196 2017-10-11 17:42:35,936 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node5 checking its request amount 2017-10-11 17:42:35,936 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 204 to run in 60 seconds 2017-10-11 17:42:35,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 25 2017-10-11 17:42:35,936 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:42:35,936 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 26 to run in 60 seconds 2017-10-11 17:42:35,972 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 197 2017-10-11 17:42:35,973 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 205 to run in 60 seconds 2017-10-11 17:42:36,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-11 17:42:36,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:42:36,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:42:36,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node5 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:42:36,388 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:42:36,820 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 203 2017-10-11 17:42:36,821 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:42:36,821 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 206 to run in 10 seconds 2017-10-11 17:42:42,477 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node5.log 2017-10-11 17:42:42,478 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-10-11 17:42:43,540 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-11 17:42:43,554 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-11 17:42:43,589 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-11 17:42:43,590 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-11 17:42:44,207 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-11 17:42:44,207 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-10-11 17:42:44,238 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:42:44,244 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 16 2017-10-11 17:42:44,245 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0017678621225059032 seconds 2017-10-11 17:42:44,262 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node5 2017-10-11 17:42:44,291 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:42:44,300 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 5 2017-10-11 17:42:44,300 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0005046389997005463 seconds 2017-10-11 17:42:44,347 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:42:44,347 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:42:44,347 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:42:44,347 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:42:44,347 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-11 17:42:44,347 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-11 17:42:44,348 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-11 17:42:44,348 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-11 17:42:44,349 | DEBUG | replica.py ( 313) | h | Node5:0 set watermarks as 0 300 2017-10-11 17:42:44,349 | DISPLAY | replicas.py ( 36) | grow | Node5 added replica Node5:0 to instance 0 (master) 2017-10-11 17:42:44,349 | DEBUG | replica.py ( 313) | h | Node5:1 set watermarks as 0 300 2017-10-11 17:42:44,350 | DISPLAY | replicas.py ( 36) | grow | Node5 added replica Node5:1 to instance 1 (backup) 2017-10-11 17:42:44,350 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-11 17:42:44,350 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-11 17:42:44,350 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-11 17:42:44,350 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-11 17:42:44,350 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-11 17:42:44,350 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-11 17:42:44,351 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-10-11 17:42:44,374 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:42:44,381 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 11 2017-10-11 17:42:44,382 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0006881179288029671 seconds 2017-10-11 17:42:44,389 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node5 has no upgrade events 2017-10-11 17:42:44,390 | INFO | node.py (2420) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2017-10-11 17:42:44,392 | DEBUG | motor.py ( 34) | set_status | Node5 changing status from stopped to starting 2017-10-11 17:42:44,392 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:42:44,392 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:42:44,393 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:42:44,393 | DEBUG | zstack.py ( 319) | start | Node5 starting with restricted as True and reSetupAuth as True 2017-10-11 17:42:44,393 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-11 17:42:44,394 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:42:44,394 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node5/public_keys] 2017-10-11 17:42:44,395 | DEBUG | zstack.py ( 347) | open | Node5 will bind its listener at 9701 2017-10-11 17:42:44,395 | INFO | stacks.py ( 84) | start | CONNECTION: Node5 listening for other nodes at 0.0.0.0:9701 2017-10-11 17:42:44,396 | DEBUG | zstack.py ( 319) | start | Node5C starting with restricted as False and reSetupAuth as True 2017-10-11 17:42:44,397 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-11 17:42:44,397 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:42:44,397 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-11 17:42:44,398 | DEBUG | zstack.py ( 347) | open | Node5C will bind its listener at 9702 2017-10-11 17:42:44,398 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-10-11 17:42:44,398 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-10-11 17:42:44,399 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-10-11 17:42:44,399 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-10-11 17:42:44,399 | INFO | node.py ( 621) | start | Node5 first time running... 2017-10-11 17:42:44,400 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node5 found the following missing connections: Node3, Node4, Node2, Node1 2017-10-11 17:42:44,401 | TRACE | remote.py ( 86) | connect | connecting socket 78 27817200 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:42:44,402 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node3 at 10.0.0.4:9705 2017-10-11 17:42:44,404 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node3 2017-10-11 17:42:44,405 | TRACE | remote.py ( 86) | connect | connecting socket 81 27838592 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:42:44,405 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node4 at 10.0.0.5:9707 2017-10-11 17:42:44,405 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node4 2017-10-11 17:42:44,409 | TRACE | remote.py ( 86) | connect | connecting socket 84 27843248 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-11 17:42:44,410 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node2 at 10.0.0.3:9703 2017-10-11 17:42:44,410 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node2 2017-10-11 17:42:44,411 | TRACE | remote.py ( 86) | connect | connecting socket 85 28023680 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:42:44,411 | INFO | zstack.py ( 590) | connect | CONNECTION: Node5 looking for Node1 at 10.0.0.2:9701 2017-10-11 17:42:44,411 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 pinged Node1 2017-10-11 17:42:44,412 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node5 next check for retries in 15.00 seconds 2017-10-11 17:42:44,423 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node1 2017-10-11 17:42:44,424 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node1 2017-10-11 17:42:44,424 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node4 2017-10-11 17:42:44,424 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node4 2017-10-11 17:42:44,425 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node2 2017-10-11 17:42:44,425 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node2 2017-10-11 17:42:44,425 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'pi' to Node3 2017-10-11 17:42:44,425 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'pi' to Node3 2017-10-11 17:42:44,477 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:44,477 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node2 2017-10-11 17:42:44,479 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from set() to {'Node2'} 2017-10-11 17:42:44,479 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node2 2017-10-11 17:42:44,480 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} to 1 recipients: ['Node2'] 2017-10-11 17:42:44,480 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:44,481 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node1 2017-10-11 17:42:44,482 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}' to Node2 2017-10-11 17:42:44,483 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}' to Node2 2017-10-11 17:42:44,483 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from {'Node2'} to {'Node2', 'Node1'} 2017-10-11 17:42:44,483 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node1 2017-10-11 17:42:44,484 | DEBUG | motor.py ( 34) | set_status | Node5 changing status from starting to started_hungry 2017-10-11 17:42:44,484 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node1'} 2017-10-11 17:42:44,484 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node5 has no ViewChangeDone message to send for view 0 2017-10-11 17:42:44,485 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node1 2017-10-11 17:42:44,485 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node1'] 2017-10-11 17:42:44,485 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} to 1 recipients: ['Node1'] 2017-10-11 17:42:44,496 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node1 into one transmission 2017-10-11 17:42:44,496 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}']) 2017-10-11 17:42:44,496 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' 2017-10-11 17:42:44,497 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' to Node1 2017-10-11 17:42:44,508 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:44,509 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node4 2017-10-11 17:42:44,510 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from {'Node2', 'Node1'} to {'Node2', 'Node4', 'Node1'} 2017-10-11 17:42:44,510 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node4 2017-10-11 17:42:44,510 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node4', 'Node1'} 2017-10-11 17:42:44,510 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node5 has no ViewChangeDone message to send for view 0 2017-10-11 17:42:44,510 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node4 2017-10-11 17:42:44,511 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-10-11 17:42:44,511 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} to 1 recipients: ['Node4'] 2017-10-11 17:42:44,512 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node4 into one transmission 2017-10-11 17:42:44,512 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}']) 2017-10-11 17:42:44,513 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' 2017-10-11 17:42:44,513 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' to Node4 2017-10-11 17:42:44,533 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:44,535 | DEBUG | zstack.py ( 669) | handlePingPong | Node5 got pong from Node3 2017-10-11 17:42:44,536 | DEBUG | keep_in_touch.py ( 68) | conns | Node5's connections changed from {'Node2', 'Node4', 'Node1'} to {'Node2', 'Node4', 'Node1', 'Node3'} 2017-10-11 17:42:44,536 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node5 now connected to Node3 2017-10-11 17:42:44,536 | DEBUG | motor.py ( 34) | set_status | Node5 changing status from started_hungry to started 2017-10-11 17:42:44,536 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 5 and nodes {'Node2', 'Node4', 'Node1', 'Node3'} 2017-10-11 17:42:44,536 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node5 has no ViewChangeDone message to send for view 0 2017-10-11 17:42:44,537 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node5 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-10-11 17:42:44,537 | DEBUG | node.py (2608) | send | Node5 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-10-11 17:42:44,537 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} to 1 recipients: ['Node3'] 2017-10-11 17:42:44,538 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:42:44,538 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}']) 2017-10-11 17:42:44,538 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' 2017-10-11 17:42:44,540 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":5}"]}' to Node3 2017-10-11 17:42:44,687 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:44,688 | DEBUG | zstack.py ( 663) | handlePingPong | Node5C got ping from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:44,688 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'po' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:44,689 | DEBUG | zstack.py ( 645) | sendPingPong | Node5C ponged b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:44,706 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:44,706 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} 2017-10-11 17:42:44,707 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} 2017-10-11 17:42:44,707 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': None, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': None} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:44,707 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:44,708 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ppSeqNo":null,"viewNo":null,"txnSeqNo":5}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:44,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:44,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:47,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action dump_json_file with id 4 2017-10-11 17:42:48,986 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5C got 1 messages through listener 2017-10-11 17:42:48,987 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: BATCH{'signature': None, 'messages': ['{"reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","identifier":"XhYtvJqezMUKfF6KVNaGmT"}', '{"reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","identifier":"XhYtvJqezMUKfF6KVNaGmT"}']} 2017-10-11 17:42:48,989 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743693883477 2017-10-11 17:42:48,989 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,990 | DEBUG | node.py (2236) | verifySignature | Node5 authenticated XhYtvJqezMUKfF6KVNaGmT signature on request 1507743707150215 2017-10-11 17:42:48,990 | TRACE | node.py (1388) | validateClientMsg | Node5C received CLIENT message: SafeRequest: {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,991 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,991 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:48,991 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,991 | INFO | propagator.py ( 148) | propagate | Node5 propagating request ('XhYtvJqezMUKfF6KVNaGmT', 1507743693883477) from client b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:48,991 | DEBUG | node.py (2608) | send | Node5 sending message PROPAGATE{'request': {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} to all recipients: ['Node1', 'Node4', 'Node2', 'Node3'] 2017-10-11 17:42:48,991 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'reqId': 1507743693883477, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT7'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since not finalised 2017-10-11 17:42:48,992 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"op":"REQACK","reqId":1507743693883477,"identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:48,992 | DEBUG | node.py (1434) | processClientInBox | Node5C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,992 | DEBUG | node.py (1708) | processRequest | Node5 received client request: SafeRequest: {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:48,992 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} 2017-10-11 17:42:48,992 | INFO | propagator.py ( 148) | propagate | Node5 propagating request ('XhYtvJqezMUKfF6KVNaGmT', 1507743707150215) from client b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:42:48,993 | DEBUG | node.py (2608) | send | Node5 sending message PROPAGATE{'request': {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'}, 'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a'} to all recipients: ['Node1', 'Node4', 'Node2', 'Node3'] 2017-10-11 17:42:48,993 | DEBUG | propagator.py ( 246) | tryForwarding | Node5 not forwarding request SafeRequest: {'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'reqId': 1507743707150215, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT8'}, 'identifier': 'XhYtvJqezMUKfF6KVNaGmT'} to its replicas since not finalised 2017-10-11 17:42:48,994 | TRACE | zstack.py ( 754) | transmitThroughListener | Node5C transmitting b'{"op":"REQACK","reqId":1507743707150215,"identifier":"XhYtvJqezMUKfF6KVNaGmT"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:42:48,994 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node1 into one transmission 2017-10-11 17:42:48,994 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"request":{"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}', b'{"request":{"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}']) 2017-10-11 17:42:48,995 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' 2017-10-11 17:42:48,995 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' to Node1 2017-10-11 17:42:48,995 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node4 into one transmission 2017-10-11 17:42:48,995 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"request":{"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}', b'{"request":{"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}']) 2017-10-11 17:42:48,996 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' 2017-10-11 17:42:48,996 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' to Node4 2017-10-11 17:42:48,996 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node2 into one transmission 2017-10-11 17:42:48,996 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"request":{"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}', b'{"request":{"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}']) 2017-10-11 17:42:48,996 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' 2017-10-11 17:42:48,996 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' to Node2 2017-10-11 17:42:48,997 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:42:48,997 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"request":{"signature":"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3","reqId":1507743693883477,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT7"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}', b'{"request":{"signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","reqId":1507743707150215,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT8"},"identifier":"XhYtvJqezMUKfF6KVNaGmT"},"op":"PROPAGATE","senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a"}']) 2017-10-11 17:42:48,997 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' 2017-10-11 17:42:48,997 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"request\\":{\\"signature\\":\\"54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3\\",\\"reqId\\":1507743693883477,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT7\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}","{\\"request\\":{\\"signature\\":\\"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD\\",\\"reqId\\":1507743707150215,\\"operation\\":{\\"type\\":\\"1\\",\\"dest\\":\\"542MVr22zcHbVyGzaXmbT8\\"},\\"identifier\\":\\"XhYtvJqezMUKfF6KVNaGmT\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"+>]}>OKfYQDWB?@taDN+?+{nc\\\\\\/ZoKRv9iH*\\\\\\/^F$a\\"}"]}' to Node3 2017-10-11 17:42:54,358 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node5 running action checkPerformance with id 1 2017-10-11 17:42:54,358 | TRACE | node.py (1978) | checkPerformance | Node5 checking its performance 2017-10-11 17:42:54,358 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-11 17:42:56,562 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:42:56,563 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node3 2017-10-11 17:42:56,563 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node3 2017-10-11 17:42:56,563 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node2 2017-10-11 17:42:56,564 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node2 2017-10-11 17:42:56,564 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node2 2017-10-11 17:42:56,564 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node2 2017-10-11 17:42:56,564 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node3 2017-10-11 17:42:56,564 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node3 2017-10-11 17:42:56,572 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:42:56,572 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":18,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"ppSeqNo":8}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}']} 2017-10-11 17:42:56,573 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":18,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"ppSeqNo":8}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}']}, 'Node3') 2017-10-11 17:42:56,573 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"viewNo":4,"op":"CURRENT_STATE"}', '{"txnSeqNo":5,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"ppSeqNo":6}', '{"txnSeqNo":18,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"ppSeqNo":8}', '{"txnSeqNo":11,"viewNo":4,"op":"LEDGER_STATUS","merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"ppSeqNo":2}']} 2017-10-11 17:42:56,573 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,573 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'viewNo': 4, 'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}]}, 'Node3') 2017-10-11 17:42:56,574 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,574 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,574 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'viewNo': 4, 'ledgerId': 0, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'txnSeqNo': 5}, 'Node3') 2017-10-11 17:42:56,574 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,574 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,574 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'op': 'LEDGER_STATUS', 'viewNo': 4, 'ledgerId': 1, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'txnSeqNo': 18}, 'Node3') 2017-10-11 17:42:56,575 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,575 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,575 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'viewNo': 4, 'ledgerId': 2, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'txnSeqNo': 11}, 'Node3') 2017-10-11 17:42:56,575 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,575 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node4 2017-10-11 17:42:56,575 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node4 2017-10-11 17:42:56,576 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} from Node3 2017-10-11 17:42:56,576 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node5 stashing a message for a future view: VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,577 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} from Node3 2017-10-11 17:42:56,577 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,577 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,577 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node3 2017-10-11 17:42:56,578 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node3 2017-10-11 17:42:56,578 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node3 2017-10-11 17:42:56,578 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node3 2017-10-11 17:42:56,579 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'po' to Node4 2017-10-11 17:42:56,579 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'po' to Node4 2017-10-11 17:42:56,580 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:56,584 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'signature': None, 'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":18,"ppSeqNo":8,"ledgerId":1,"viewNo":4,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}']} 2017-10-11 17:42:56,584 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":18,"ppSeqNo":8,"ledgerId":1,"viewNo":4,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}']}, 'Node2') 2017-10-11 17:42:56,584 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":4,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"txnSeqNo":5,"ppSeqNo":6,"ledgerId":0,"viewNo":4,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","op":"LEDGER_STATUS"}', '{"txnSeqNo":18,"ppSeqNo":8,"ledgerId":1,"viewNo":4,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ppSeqNo":2,"ledgerId":2,"viewNo":4,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","op":"LEDGER_STATUS"}']} 2017-10-11 17:42:56,585 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,585 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4}, 'Node2') 2017-10-11 17:42:56,585 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,585 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,585 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node2') 2017-10-11 17:42:56,585 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,585 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,586 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 1, 'txnSeqNo': 18, 'viewNo': 4, 'ppSeqNo': 8, 'op': 'LEDGER_STATUS', 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}, 'Node2') 2017-10-11 17:42:56,586 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,586 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,586 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node2') 2017-10-11 17:42:56,586 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,586 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} from Node2 2017-10-11 17:42:56,587 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node5 stashing a message for a future view: VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,611 | INFO | node.py (1967) | _start_view_change_if_possible | VIEW CHANGE: Node5 starting view change for 4 after 2 view change indications from other nodes 2017-10-11 17:42:56,611 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action _check_view_change_completed with id 7 to run in 60 seconds 2017-10-11 17:42:56,611 | DEBUG | replica.py ( 455) | on_view_change_start | Node5:0 setting last prepared for master to None 2017-10-11 17:42:56,611 | DEBUG | node.py (2146) | startViewChange | Node5 resetting monitor stats after view change 2017-10-11 17:42:56,611 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:42:56,612 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}, 'Node3') 2017-10-11 17:42:56,612 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}, 'Node2') 2017-10-11 17:42:56,612 | DEBUG | node.py ( 996) | processStashedMsgsForView | Node5 processed 2 stashed msgs for view no 4 2017-10-11 17:42:56,613 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node5 changed to view 4, will start catchup now 2017-10-11 17:42:56,614 | DEBUG | node.py (2185) | start_catchup | Node5 does not start the catchup procedure because it is already in this state 2017-10-11 17:42:56,614 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} from Node2 2017-10-11 17:42:56,614 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,616 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,616 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node2 2017-10-11 17:42:56,617 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node2 2017-10-11 17:42:56,617 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node2 2017-10-11 17:42:56,617 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node2 2017-10-11 17:42:56,617 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,618 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node5 needs 1 ViewChangeDone messages 2017-10-11 17:42:56,618 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:42:56,618 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,619 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node5 got view change quorum (2 >= 2) 2017-10-11 17:42:56,620 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node5 has not received ViewChangeDone from the next primary Node5 (viewNo: 4, totalNodes: 5) 2017-10-11 17:42:56,620 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:42:56,621 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:42:56,622 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":18,"ppSeqNo":8}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}']} 2017-10-11 17:42:56,625 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":18,"ppSeqNo":8}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}']}, 'Node4') 2017-10-11 17:42:56,627 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}]}', '{"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":5,"ppSeqNo":6}', '{"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":18,"ppSeqNo":8}', '{"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"viewNo":4,"op":"LEDGER_STATUS","txnSeqNo":11,"ppSeqNo":2}']} 2017-10-11 17:42:56,627 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,628 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4}, 'Node4') 2017-10-11 17:42:56,628 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,632 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,634 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'ledgerId': 0, 'viewNo': 4, 'txnSeqNo': 5}, 'Node4') 2017-10-11 17:42:56,635 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,636 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,637 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'op': 'LEDGER_STATUS', 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'ledgerId': 1, 'viewNo': 4, 'txnSeqNo': 18}, 'Node4') 2017-10-11 17:42:56,637 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,637 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,638 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'ledgerId': 2, 'viewNo': 4, 'txnSeqNo': 11}, 'Node4') 2017-10-11 17:42:56,638 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,638 | DEBUG | zstack.py ( 663) | handlePingPong | Node5 got ping from Node1 2017-10-11 17:42:56,639 | DEBUG | zstack.py ( 645) | sendPingPong | Node5 ponged Node1 2017-10-11 17:42:56,639 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} from Node4 2017-10-11 17:42:56,640 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}, 'Node4') 2017-10-11 17:42:56,640 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} from Node4 2017-10-11 17:42:56,641 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,641 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,641 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node2', 'Node3'} that its ledger of type 0 is latest 2017-10-11 17:42:56,641 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node5 found from ledger status LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} that it does not need catchup 2017-10-11 17:42:56,642 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:42:56,642 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,642 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:42:56,642 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,643 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:42:56,643 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:42:56,643 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:42:56,644 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 0 2017-10-11 17:42:56,644 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,644 | DEBUG | node.py ( 941) | checkInstances | Node5 choosing to start election on the basis of count 5 and nodes {'Node2', 'Node4', 'Node1', 'Node3'} 2017-10-11 17:42:56,645 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-10-11 17:42:56,645 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 2 2017-10-11 17:42:56,645 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,646 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 2 2017-10-11 17:42:56,646 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node3'] 2017-10-11 17:42:56,646 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 2 2017-10-11 17:42:56,648 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-10-11 17:42:56,648 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 2 2017-10-11 17:42:56,648 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node1', 'Node4', 'Node2', 'Node3'] 2017-10-11 17:42:56,649 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 2 2017-10-11 17:42:56,649 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 2 stashed ledger statuses for ledger 2 2017-10-11 17:42:56,650 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node3 2017-10-11 17:42:56,650 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,650 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,651 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node2 2017-10-11 17:42:56,651 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,652 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,652 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 0, caught up 0 in total 2017-10-11 17:42:56,652 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node4 2017-10-11 17:42:56,653 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node5 stashing ledger status LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node4 2017-10-11 17:42:56,653 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node4 2017-10-11 17:42:56,653 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,654 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,654 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,654 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node5 found from ledger status LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} that it does not need catchup 2017-10-11 17:42:56,654 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:42:56,655 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,655 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:42:56,655 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,656 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:42:56,656 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:42:56,656 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:42:56,657 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 2 2017-10-11 17:42:56,657 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,657 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-11 17:42:56,657 | DEBUG | upgrader.py ( 208) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc processing config ledger for any upgrades 2017-10-11 17:42:56,658 | INFO | upgrader.py ( 214) | processLedger | 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:42:56,659 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node5' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:42:56,659 | INFO | upgrader.py ( 304) | handleUpgradeTxn | Node 'Node5' disregards upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:42:56,661 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-10-11 17:42:56,661 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node1 for ledger status of ledger 1 2017-10-11 17:42:56,662 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,662 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node2 for ledger status of ledger 1 2017-10-11 17:42:56,662 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node3'] 2017-10-11 17:42:56,663 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node3 for ledger status of ledger 1 2017-10-11 17:42:56,663 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-10-11 17:42:56,664 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node4 for ledger status of ledger 1 2017-10-11 17:42:56,664 | DEBUG | node.py (2608) | send | Node5 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node1', 'Node4', 'Node2', 'Node3'] 2017-10-11 17:42:56,664 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node5 asking Node5 for ledger status of ledger 1 2017-10-11 17:42:56,665 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node5 going to process 3 stashed ledger statuses for ledger 1 2017-10-11 17:42:56,665 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node3 2017-10-11 17:42:56,666 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,666 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,666 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node3'] 2017-10-11 17:42:56,667 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node2 2017-10-11 17:42:56,667 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,667 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,668 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,668 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node4 2017-10-11 17:42:56,668 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,669 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,669 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node4'] 2017-10-11 17:42:56,669 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node5' last upgrade txn is None 2017-10-11 17:42:56,670 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node5 completed catching up ledger 2, caught up 0 in total 2017-10-11 17:42:56,671 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,671 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node5 got view change quorum (3 >= 2) 2017-10-11 17:42:56,671 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node5 has not received ViewChangeDone from the next primary Node5 (viewNo: 4, totalNodes: 5) 2017-10-11 17:42:56,672 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:42:56,672 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node1 into one transmission 2017-10-11 17:42:56,672 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'po', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']) 2017-10-11 17:42:56,673 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node1: b'{"signature":null,"op":"BATCH","messages":["po","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' 2017-10-11 17:42:56,673 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["po","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' to Node1 2017-10-11 17:42:56,674 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node4 into one transmission 2017-10-11 17:42:56,675 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}']) 2017-10-11 17:42:56,676 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node4: b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' 2017-10-11 17:42:56,676 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' to Node4 2017-10-11 17:42:56,677 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node2 into one transmission 2017-10-11 17:42:56,677 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}']) 2017-10-11 17:42:56,678 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' 2017-10-11 17:42:56,678 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' to Node2 2017-10-11 17:42:56,680 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 5 msgs to Node3 into one transmission 2017-10-11 17:42:56,680 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}']) 2017-10-11 17:42:56,681 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' 2017-10-11 17:42:56,682 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' to Node3 2017-10-11 17:42:56,695 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 3 messages through listener 2017-10-11 17:42:56,696 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']} 2017-10-11 17:42:56,698 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']}, 'Node3') 2017-10-11 17:42:56,699 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"msg_type":"LEDGER_STATUS"}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']} 2017-10-11 17:42:56,700 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,700 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-10-11 17:42:56,701 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,701 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,702 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-10-11 17:42:56,702 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,702 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,703 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-10-11 17:42:56,703 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,703 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,704 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS'}, 'Node3') 2017-10-11 17:42:56,704 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,704 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,705 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node3') 2017-10-11 17:42:56,705 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,705 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}']} 2017-10-11 17:42:56,706 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}']}, 'Node2') 2017-10-11 17:42:56,707 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}']} 2017-10-11 17:42:56,708 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,708 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node2') 2017-10-11 17:42:56,709 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,709 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,709 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node2') 2017-10-11 17:42:56,710 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,710 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,710 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node2') 2017-10-11 17:42:56,711 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,711 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,712 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node2') 2017-10-11 17:42:56,712 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,712 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,713 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node2') 2017-10-11 17:42:56,713 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,713 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":18,"ledgerId":1,"viewNo":4,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"seqNoStart":16,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":8}']} 2017-10-11 17:42:56,714 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":18,"ledgerId":1,"viewNo":4,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"seqNoStart":16,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":8}']}, 'Node4') 2017-10-11 17:42:56,715 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"seqNoEnd":18,"ledgerId":1,"viewNo":4,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"seqNoStart":16,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":8}']} 2017-10-11 17:42:56,715 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,716 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node4') 2017-10-11 17:42:56,716 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,717 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,717 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node4') 2017-10-11 17:42:56,717 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,718 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,718 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node4') 2017-10-11 17:42:56,719 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,719 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,719 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node4') 2017-10-11 17:42:56,720 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,720 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node4: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,720 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'seqNoEnd': 18, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoStart': 16, 'viewNo': 4}, 'Node4') 2017-10-11 17:42:56,721 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,721 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node3 2017-10-11 17:42:56,722 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,722 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,723 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node3 2017-10-11 17:42:56,723 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,723 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,724 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node3 2017-10-11 17:42:56,724 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,724 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,725 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node3'] 2017-10-11 17:42:56,725 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node3 2017-10-11 17:42:56,725 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,726 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,726 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node3'] 2017-10-11 17:42:56,727 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node3 2017-10-11 17:42:56,727 | DEBUG | ledger_manager.py ( 706) | canStartCatchUpProcess | Node5 cannot start catchup since received only 1 consistency proofs but need at least 2 2017-10-11 17:42:56,728 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node2 2017-10-11 17:42:56,728 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,729 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,729 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node2 2017-10-11 17:42:56,730 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,730 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,730 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node2 2017-10-11 17:42:56,731 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,731 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,731 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,732 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node2 2017-10-11 17:42:56,733 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,733 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,733 | DEBUG | node.py (2608) | send | Node5 sending message LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': None, 'txnSeqNo': 16, 'merkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'viewNo': None} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,734 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node2 2017-10-11 17:42:56,734 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_CPs_if_needed) with id 1 to run in 20 seconds 2017-10-11 17:42:56,735 | DEBUG | ledger_manager.py ( 688) | canStartCatchUpProcess | Node5 deciding on the basis of CPs {'Node2': CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node3': CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}} and f 1 2017-10-11 17:42:56,735 | DEBUG | node.py (1479) | preLedgerCatchUp | Node5 going to process any ordered requests before starting catchup. 2017-10-11 17:42:56,736 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,736 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:42:56,736 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node5:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:42:56,736 | DEBUG | node.py (1888) | force_process_ordered | Node5 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:42:56,737 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node5 processed 0 stashed ordered requests 2017-10-11 17:42:56,737 | DEBUG | monitor.py ( 192) | reset | Node5's Monitor being reset 2017-10-11 17:42:56,737 | INFO | node.py (1489) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 1 2017-10-11 17:42:56,738 | DEBUG | ledger_manager.py ( 788) | startCatchUpProcess | Node5 started catching up with consistency proof CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ('C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW',), 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,738 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'catchupTill': 18, 'seqNoStart': 17, 'seqNoEnd': 17} to 1 recipients: ['Node1'] 2017-10-11 17:42:56,739 | DEBUG | node.py (2608) | send | Node5 sending message CATCHUP_REQ{'ledgerId': 1, 'catchupTill': 18, 'seqNoStart': 18, 'seqNoEnd': 18} to 1 recipients: ['Node2'] 2017-10-11 17:42:56,739 | TRACE | has_action_queue.py ( 36) | _schedule | Node5 scheduling action partial(request_txns_if_needed) with id 2 to run in 10.2 seconds 2017-10-11 17:42:56,739 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node4 2017-10-11 17:42:56,740 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,740 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,740 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,741 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node4 2017-10-11 17:42:56,741 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,741 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,742 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,742 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node4 2017-10-11 17:42:56,743 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,743 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,743 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node4 2017-10-11 17:42:56,744 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,744 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,744 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node4 2017-10-11 17:42:56,744 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:42:56,745 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"ledgerId":1,"catchupTill":18,"seqNoStart":17,"op":"CATCHUP_REQ","seqNoEnd":17}' to Node1 2017-10-11 17:42:56,745 | TRACE | batched.py ( 96) | flushOutBoxes | Node5 sending msg b'{"ledgerId":1,"catchupTill":18,"seqNoStart":17,"op":"CATCHUP_REQ","seqNoEnd":17}' to Node1 2017-10-11 17:42:56,745 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 3 msgs to Node2 into one transmission 2017-10-11 17:42:56,746 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}', b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}', b'{"ledgerId":1,"catchupTill":18,"seqNoStart":18,"op":"CATCHUP_REQ","seqNoEnd":18}']) 2017-10-11 17:42:56,746 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"catchupTill\\":18,\\"seqNoStart\\":18,\\"op\\":\\"CATCHUP_REQ\\",\\"seqNoEnd\\":18}"]}' 2017-10-11 17:42:56,746 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"catchupTill\\":18,\\"seqNoStart\\":18,\\"op\\":\\"CATCHUP_REQ\\",\\"seqNoEnd\\":18}"]}' to Node2 2017-10-11 17:42:56,746 | DEBUG | batched.py ( 100) | flushOutBoxes | Node5 batching 2 msgs to Node3 into one transmission 2017-10-11 17:42:56,747 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}', b'{"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","ppSeqNo":null,"viewNo":null,"txnSeqNo":16}']) 2017-10-11 17:42:56,747 | TRACE | batched.py ( 110) | flushOutBoxes | Node5 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' 2017-10-11 17:42:56,748 | DEBUG | zstack.py ( 723) | transmit | Node5 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L\\",\\"ppSeqNo\\":null,\\"viewNo\\":null,\\"txnSeqNo\\":16}"]}' to Node3 2017-10-11 17:42:56,749 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:42:56,750 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}']} 2017-10-11 17:42:56,750 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}']}, 'Node1') 2017-10-11 17:42:56,751 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ppSeqNo":2,"txnSeqNo":11,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ppSeqNo":8,"txnSeqNo":18,"viewNo":4},"op":"MESSAGE_RESPONSE"}']} 2017-10-11 17:42:56,751 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,751 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node1') 2017-10-11 17:42:56,751 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,751 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,751 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}}, 'Node1') 2017-10-11 17:42:56,752 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'viewNo': 4, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}} 2017-10-11 17:42:56,752 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,752 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node1') 2017-10-11 17:42:56,752 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,752 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,752 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}}, 'Node1') 2017-10-11 17:42:56,752 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'viewNo': 4, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}} 2017-10-11 17:42:56,753 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: BATCH{'signature': None, 'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":18,"ppSeqNo":8,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}']} 2017-10-11 17:42:56,753 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":18,"ppSeqNo":8,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}']}, 'Node1') 2017-10-11 17:42:56,753 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":4,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,9,"98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN"]],"name":"Node1","viewNo":4}],"op":"CURRENT_STATE"}', '{"viewNo":4,"txnSeqNo":5,"ppSeqNo":6,"merkleRoot":"3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7","ledgerId":0,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":18,"ppSeqNo":8,"merkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"op":"LEDGER_STATUS"}', '{"viewNo":4,"txnSeqNo":11,"ppSeqNo":2,"merkleRoot":"dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F","ledgerId":2,"op":"LEDGER_STATUS"}']} 2017-10-11 17:42:56,753 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,753 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4}, 'Node1') 2017-10-11 17:42:56,753 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} 2017-10-11 17:42:56,754 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,754 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 0, 'txnSeqNo': 5, 'viewNo': 4, 'ppSeqNo': 6, 'op': 'LEDGER_STATUS', 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7'}, 'Node1') 2017-10-11 17:42:56,754 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} 2017-10-11 17:42:56,754 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,755 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 1, 'txnSeqNo': 18, 'viewNo': 4, 'ppSeqNo': 8, 'op': 'LEDGER_STATUS', 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM'}, 'Node1') 2017-10-11 17:42:56,755 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} 2017-10-11 17:42:56,755 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,755 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ledgerId': 2, 'txnSeqNo': 11, 'viewNo': 4, 'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F'}, 'Node1') 2017-10-11 17:42:56,755 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} 2017-10-11 17:42:56,755 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node1 2017-10-11 17:42:56,756 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,756 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,756 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node1', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,756 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node1 2017-10-11 17:42:56,757 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,757 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,757 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node1', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,757 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node1 2017-10-11 17:42:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,758 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node1 2017-10-11 17:42:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,758 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,759 | DEBUG | node.py ( 907) | process_current_state_message | Node5 processing current state CURRENT_STATE{'primary': [{'viewNo': 4, 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}], 'viewNo': 4} from Node1 2017-10-11 17:42:56,759 | DEBUG | node.py (1205) | sendToElector | Node5 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]}, 'Node1') 2017-10-11 17:42:56,759 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'ppSeqNo': 6, 'txnSeqNo': 5, 'merkleRoot': '3a2yaKvUMW63AyrxnDLq6c4G7AQCx2u32ZL8PcWes7n7', 'viewNo': 4} from Node1 2017-10-11 17:42:56,760 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,760 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 0 of size 5 with 5 2017-10-11 17:42:56,760 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'ppSeqNo': 8, 'txnSeqNo': 18, 'merkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'viewNo': 4} from Node1 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 1 of size 16 with 18 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node5 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'ppSeqNo': 2, 'txnSeqNo': 11, 'merkleRoot': 'dR7dm9mURwRfqNyHg3i5ymsZHKpSgJmqwNUw83DZG4F', 'viewNo': 4} from Node1 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node5 comparing its ledger 2 of size 11 with 11 2017-10-11 17:42:56,761 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node5 found out from {'Node4', 'Node1', 'Node2', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:42:56,762 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node5's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 4, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 9, '98hFgstpP3hBqogWWjGAAao2kkUXTCuXrhgWkwugSfJN']]} 2017-10-11 17:42:56,762 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node5 got view change quorum (4 >= 2) 2017-10-11 17:42:56,763 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node5 has not received ViewChangeDone from the next primary Node5 (viewNo: 4, totalNodes: 5) 2017-10-11 17:42:56,763 | DEBUG | primary_selector.py ( 266) | _startSelection | Node5 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:42:56,764 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 2 messages through listener 2017-10-11 17:42:56,765 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: BATCH{'signature': None, 'messages': ['{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"consProof":[],"ledgerId":1,"txns":{"18":{"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","txnTime":1507743707,"reqId":1507743707150215,"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"}},"op":"CATCHUP_REP"}']} 2017-10-11 17:42:56,765 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"consProof":[],"ledgerId":1,"txns":{"18":{"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","txnTime":1507743707,"reqId":1507743707150215,"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"}},"op":"CATCHUP_REP"}']}, 'Node2') 2017-10-11 17:42:56,765 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","ledgerId":1,"seqNoEnd":18,"viewNo":4,"ppSeqNo":8,"seqNoStart":16,"hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"op":"CONSISTENCY_PROOF","oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L"}', '{"consProof":[],"ledgerId":1,"txns":{"18":{"identifier":"XhYtvJqezMUKfF6KVNaGmT","signature":"3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD","txnTime":1507743707,"reqId":1507743707150215,"dest":"542MVr22zcHbVyGzaXmbT8","type":"1"}},"op":"CATCHUP_REP"}']} 2017-10-11 17:42:56,765 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,766 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node2') 2017-10-11 17:42:56,766 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,766 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,766 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node2') 2017-10-11 17:42:56,766 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,767 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node2: CATCHUP_REP{'ledgerId': 1, 'consProof': [], 'txns': {'18': {'type': '1', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'dest': '542MVr22zcHbVyGzaXmbT8', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'txnTime': 1507743707, 'reqId': 1507743707150215}}} 2017-10-11 17:42:56,767 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'consProof': [], 'ledgerId': 1, 'op': 'CATCHUP_REP', 'txns': {'18': {'type': '1', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'dest': '542MVr22zcHbVyGzaXmbT8', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'txnTime': 1507743707, 'reqId': 1507743707150215}}}, 'Node2') 2017-10-11 17:42:56,767 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CATCHUP_REP{'ledgerId': 1, 'consProof': [], 'txns': {'18': {'type': '1', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'dest': '542MVr22zcHbVyGzaXmbT8', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'txnTime': 1507743707, 'reqId': 1507743707150215}}} 2017-10-11 17:42:56,767 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: BATCH{'signature': None, 'messages': ['{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']} 2017-10-11 17:42:56,767 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']}, 'Node3') 2017-10-11 17:42:56,767 | DEBUG | node.py (1272) | unpackNodeMsg | Node5 processing a batch BATCH{'signature': None, 'messages': ['{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}', '{"ledgerId":1,"newMerkleRoot":"ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM","hashes":["C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW"],"viewNo":4,"op":"CONSISTENCY_PROOF","ppSeqNo":8,"oldMerkleRoot":"Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L","seqNoEnd":18,"seqNoStart":16}']} 2017-10-11 17:42:56,768 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,768 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node3') 2017-10-11 17:42:56,768 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,768 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node3: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,768 | DEBUG | node.py (1219) | handleOneNodeMsg | Node5 msg validated ({'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'op': 'CONSISTENCY_PROOF', 'seqNoStart': 16, 'ledgerId': 1, 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4}, 'Node3') 2017-10-11 17:42:56,768 | DEBUG | node.py (1286) | postToNodeInBox | Node5 appending to nodeInbox CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} 2017-10-11 17:42:56,769 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node2 2017-10-11 17:42:56,769 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:42:56,769 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node2 2017-10-11 17:42:56,770 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:42:56,770 | DEBUG | ledger_manager.py ( 455) | processCatchupRep | Node5 received catchup reply from Node2: CATCHUP_REP{'ledgerId': 1, 'consProof': [], 'txns': {'18': {'type': '1', 'signature': '3eUvTudLCTVAeFsqFwkbfw91X92q8vPFjpX2NSChsm6Pwo6d6ZpJno3cZaiA92MTy8RqjfxFwsXgUR5xFWtvU7CD', 'dest': '542MVr22zcHbVyGzaXmbT8', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'txnTime': 1507743707, 'reqId': 1507743707150215}}} 2017-10-11 17:42:56,770 | DEBUG | ledger_manager.py ( 460) | processCatchupRep | Node5 found 1 transactions in the catchup from Node2 2017-10-11 17:42:56,770 | DEBUG | ledger_manager.py ( 477) | processCatchupRep | Node5 merging all received catchups 2017-10-11 17:42:56,771 | DEBUG | ledger_manager.py ( 485) | processCatchupRep | Node5 merged catchups, there are 1 of them now, from 18 to 18 2017-10-11 17:42:56,771 | DEBUG | ledger_manager.py ( 493) | processCatchupRep | Node5 processed 0 catchup replies with sequence numbers [] 2017-10-11 17:42:56,771 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node3 2017-10-11 17:42:56,771 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:42:56,772 | DEBUG | ledger_manager.py ( 335) | processConsistencyProof | Node5 received consistency proof: CONSISTENCY_PROOF{'ppSeqNo': 8, 'hashes': ['C5DTXwEr8tFnFQE3WvfFUqDMM5QbcTS9HPizz3MMMfYW'], 'oldMerkleRoot': 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L', 'ledgerId': 1, 'seqNoStart': 16, 'newMerkleRoot': 'ALQxLfPiuURy7hTJHyskanjvVzix8Nz8PkaMntvjmKPM', 'seqNoEnd': 18, 'viewNo': 4} from Node3 2017-10-11 17:42:56,772 | DEBUG | ledger_manager.py ( 356) | canProcessConsistencyProof | Node5 cannot process consistency proof since ledger state is 2 2017-10-11 17:42:56,773 | TRACE | zstack.py ( 479) | _receiveFromListener | Node5 got 1 messages through listener 2017-10-11 17:42:56,774 | DEBUG | node.py (1257) | validateNodeMsg | Node5 received node message from Node1: CATCHUP_REP{'ledgerId': 1, 'consProof': ['7g3QSD2dY9HfVa5EivCGFD3TiMF3HtQGbqN315n1dS7X', 'Cp9UgZ2bY9gRegtqazEbvF1E8eTBYD5D1tqj8iqRySx5', 'Hqk4j9GFaBw44vSEABJn3nacpRHzWK6pbwrviEdqo26L'], 'txns': {'17': {'type': '1', 'signature': '54hsJM651ZVSbx6YCcqDGw1ooYBHJFY8PMnSESvWZggD1iQR97pVBrhAD9AszacVAZj4zWYqgPCdtEy23SiDGAX3', 'dest': '542MVr22zcHbVyGzaXmbT7', 'identifier': 'XhYtvJqezMUKfF6KVNaGmT', 'txnTime': 1507743