2017-08-28 15:12:33,967 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node3.log 2017-08-28 15:12:33,972 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-08-28 15:12:37,134 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-08-28 15:12:37,230 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-08-28 15:12:37,471 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-08-28 15:12:37,485 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-08-28 15:12:41,334 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-08-28 15:12:41,335 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-08-28 15:12:41,810 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:41,858 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:41,895 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.03688531200168654 seconds 2017-08-28 15:12:42,022 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node3 2017-08-28 15:12:42,074 | INFO | node.py (2378) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-08-28 15:12:42,361 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:42,447 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:42,617 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.17034363700076938 seconds 2017-08-28 15:12:42,653 | INFO | node.py (2378) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-08-28 15:12:42,967 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:42,970 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:42,971 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-08-28 15:12:42,971 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-08-28 15:12:42,971 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:42,978 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:42,979 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-08-28 15:12:42,979 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-08-28 15:12:42,979 | DEBUG | replica.py ( 311) | h | Node3:0 set watermarks as 0 300 2017-08-28 15:12:42,979 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:0 to instance 0 (master) 2017-08-28 15:12:42,980 | DEBUG | replica.py ( 311) | h | Node3:1 set watermarks as 0 300 2017-08-28 15:12:42,986 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:1 to instance 1 (backup) 2017-08-28 15:12:42,986 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-08-28 15:12:42,986 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-08-28 15:12:42,987 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-08-28 15:12:42,987 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-08-28 15:12:42,987 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-08-28 15:12:42,987 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-08-28 15:12:42,987 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-08-28 15:12:42,988 | DEBUG | node.py ( 337) | __init__ | total plugins loaded in node: 0 2017-08-28 15:12:43,064 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:43,094 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:43,188 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.09359723899979144 seconds 2017-08-28 15:12:43,229 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node3 has no upgrade events 2017-08-28 15:12:43,230 | INFO | node.py (2378) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-08-28 15:12:43,230 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from stopped to starting 2017-08-28 15:12:43,230 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:43,231 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:43,231 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:43,231 | DEBUG | zstack.py ( 319) | start | Node3 starting with restricted as True and reSetupAuth as True 2017-08-28 15:12:43,231 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-08-28 15:12:43,231 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:12:43,232 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node3/public_keys] 2017-08-28 15:12:43,318 | DEBUG | zstack.py ( 347) | open | Node3 will bind its listener at 9705 2017-08-28 15:12:43,319 | INFO | stacks.py ( 84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705 2017-08-28 15:12:43,320 | DEBUG | zstack.py ( 319) | start | Node3C starting with restricted as False and reSetupAuth as True 2017-08-28 15:12:43,320 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-08-28 15:12:43,323 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:12:43,323 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-08-28 15:12:43,330 | DEBUG | zstack.py ( 347) | open | Node3C will bind its listener at 9706 2017-08-28 15:12:43,330 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-08-28 15:12:43,331 | INFO | node.py ( 612) | start | Node3 first time running... 2017-08-28 15:12:43,340 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node3 found the following missing connections: Node2, Node4, Node1 2017-08-28 15:12:43,341 | TRACE | remote.py ( 84) | connect | connecting socket 78 43218720 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:43,351 | INFO | zstack.py ( 588) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703 2017-08-28 15:12:43,352 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 pinged Node2 2017-08-28 15:12:43,353 | TRACE | remote.py ( 84) | connect | connecting socket 80 43323184 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:43,353 | INFO | zstack.py ( 588) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707 2017-08-28 15:12:43,353 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 pinged Node4 2017-08-28 15:12:43,353 | TRACE | remote.py ( 84) | connect | connecting socket 81 43360608 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:12:43,353 | INFO | zstack.py ( 588) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701 2017-08-28 15:12:43,354 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 pinged Node1 2017-08-28 15:12:43,354 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:12:43,376 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'pi' to Node2 2017-08-28 15:12:43,376 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'pi' to Node2 2017-08-28 15:12:43,376 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'pi' to Node4 2017-08-28 15:12:43,376 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'pi' to Node4 2017-08-28 15:12:43,376 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'pi' to Node1 2017-08-28 15:12:43,376 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'pi' to Node1 2017-08-28 15:12:43,559 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:43,560 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from Node1 2017-08-28 15:12:43,563 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged Node1 2017-08-28 15:12:43,563 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from Node1 2017-08-28 15:12:43,564 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged Node1 2017-08-28 15:12:43,565 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,568 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'po', b'po']) 2017-08-28 15:12:43,568 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["po","po"],"signature":null}' 2017-08-28 15:12:43,568 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["po","po"],"signature":null}' to Node1 2017-08-28 15:12:43,568 | DEBUG | zstack.py ( 726) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,620 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:12:43,621 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from Node2 2017-08-28 15:12:43,621 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged Node2 2017-08-28 15:12:43,621 | DEBUG | zstack.py ( 667) | handlePingPong | Node3 got pong from Node1 2017-08-28 15:12:43,621 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from Node2 2017-08-28 15:12:43,621 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged Node2 2017-08-28 15:12:43,626 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,627 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'po', b'po']) 2017-08-28 15:12:43,627 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["po","po"],"signature":null}' 2017-08-28 15:12:43,645 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["po","po"],"signature":null}' to Node2 2017-08-28 15:12:43,646 | DEBUG | zstack.py ( 726) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,647 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from set() to {'Node1'} 2017-08-28 15:12:43,647 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node1 2017-08-28 15:12:43,647 | DEBUG | node.py (2566) | send | Node3 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,660 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:43,660 | DEBUG | zstack.py ( 667) | handlePingPong | Node3 got pong from Node2 2017-08-28 15:12:43,661 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}'], 'signature': None} 2017-08-28 15:12:43,661 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,661 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}'], 'signature': None} 2017-08-28 15:12:43,665 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:43,665 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node1') 2017-08-28 15:12:43,665 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:43,666 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:43,666 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'Node1') 2017-08-28 15:12:43,666 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:43,666 | DEBUG | node.py ( 889) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node1 2017-08-28 15:12:43,667 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node1 2017-08-28 15:12:43,667 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,674 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,686 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node1 2017-08-28 15:12:43,687 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to Node1 2017-08-28 15:12:43,687 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from {'Node1'} to {'Node2', 'Node1'} 2017-08-28 15:12:43,688 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node2 2017-08-28 15:12:43,688 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from starting to started_hungry 2017-08-28 15:12:43,688 | DEBUG | node.py ( 923) | checkInstances | Node3 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node1'} 2017-08-28 15:12:43,688 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node3 has no ViewChangeDone message to send for view 0 2017-08-28 15:12:43,688 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node3 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node2 2017-08-28 15:12:43,688 | DEBUG | node.py (2566) | send | Node3 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,689 | DEBUG | node.py (2566) | send | Node3 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,701 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,701 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[],"viewNo":0}', b'{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}']) 2017-08-28 15:12:43,701 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"}"],"signature":null}' 2017-08-28 15:12:43,710 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"}"],"signature":null}' to Node2 2017-08-28 15:12:43,724 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.106 seconds 2017-08-28 15:12:43,737 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:12:43,738 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}'], 'signature': None} 2017-08-28 15:12:43,738 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}'], 'signature': None}, 'Node2') 2017-08-28 15:12:43,738 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}'], 'signature': None} 2017-08-28 15:12:43,746 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:43,747 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node2') 2017-08-28 15:12:43,752 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:43,756 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:43,757 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'Node2') 2017-08-28 15:12:43,757 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:43,761 | DEBUG | node.py ( 889) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node2 2017-08-28 15:12:43,763 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node2 2017-08-28 15:12:43,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,772 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 0 is latest 2017-08-28 15:12:43,775 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} that it does not need catchup 2017-08-28 15:12:43,777 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,778 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,778 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,778 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,784 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,784 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:12:43,785 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:12:43,785 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:12:43,796 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,796 | DEBUG | node.py ( 923) | checkInstances | Node3 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node1'} 2017-08-28 15:12:43,796 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,797 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 2 2017-08-28 15:12:43,797 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,797 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 2 2017-08-28 15:12:43,797 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:12:43,797 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 2 2017-08-28 15:12:43,797 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,797 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 2 2017-08-28 15:12:43,798 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:12:43,803 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:12:43,805 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,805 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,805 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,808 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:12:43,809 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,809 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,809 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,809 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:12:43,810 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,810 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,810 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,810 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,819 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:12:43,821 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:43,822 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,822 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,822 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:12:43,823 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,823 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,823 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:12:43,823 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,837 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,837 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:12:43,837 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,838 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,838 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:12:43,838 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,838 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,838 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:12:43,838 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:43,839 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,852 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,853 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,853 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,855 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,855 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:12:43,855 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:12:43,869 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:12:43,870 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,870 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:12:43,870 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:12:43,870 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:12:43,878 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:43,878 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,878 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:12:43,878 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,879 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,879 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:12:43,879 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,890 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,890 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:12:43,891 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,891 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,891 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,896 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:12:43,896 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,897 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:12:43,897 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,897 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,897 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:12:43,897 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:43,898 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:12:43,898 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,898 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,899 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:12:43,899 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,899 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,912 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:12:43,913 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,913 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,920 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:12:43,928 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} that it does not need catchup 2017-08-28 15:12:43,928 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,928 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,928 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,928 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,928 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,928 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:12:43,928 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:12:43,929 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:12:43,929 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,929 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:12:43,929 | DEBUG | upgrader.py ( 208) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya processing config ledger for any upgrades 2017-08-28 15:12:43,929 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,930 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 1 2017-08-28 15:12:43,930 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,930 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 1 2017-08-28 15:12:43,930 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:12:43,930 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 1 2017-08-28 15:12:43,930 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,931 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 1 2017-08-28 15:12:43,931 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:12:43,944 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-08-28 15:12:43,944 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:12:43,945 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:12:43,945 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,945 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,962 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,962 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,967 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,981 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:12:43,988 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,988 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,988 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,988 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:12:43,988 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,989 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,989 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:12:43,989 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:12:43,995 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:12:43,997 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:44,015 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,015 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:44,016 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,016 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,016 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:12:44,016 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,016 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,016 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:12:44,017 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,017 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,017 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:12:44,017 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,017 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,017 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:12:44,017 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,018 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,018 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:12:44,018 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:44,018 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:12:44,033 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:12:44,037 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:12:44,038 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:12:44,061 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:44,061 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']) 2017-08-28 15:12:44,061 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:12:44,066 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:12:44,066 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:44,067 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']) 2017-08-28 15:12:44,067 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:12:44,067 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:12:44,069 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:12:44,069 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,069 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:12:44,070 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,070 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,070 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node2') 2017-08-28 15:12:44,070 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,070 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,071 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node2') 2017-08-28 15:12:44,071 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,091 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node2 2017-08-28 15:12:44,092 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,092 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,092 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node2 2017-08-28 15:12:44,093 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,093 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,098 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.362 seconds 2017-08-28 15:12:44,116 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:12:44,117 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,117 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:44,117 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:12:44,117 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,118 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node1') 2017-08-28 15:12:44,118 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,118 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,118 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node1') 2017-08-28 15:12:44,118 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:44,136 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node1 2017-08-28 15:12:44,136 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,136 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,137 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:12:44,137 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} that it does not need catchup 2017-08-28 15:12:44,137 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:12:44,137 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,137 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:44,137 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,137 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:44,137 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:12:44,138 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:12:44,138 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:12:44,138 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,138 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:12:44,138 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,138 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:12:44,138 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:12:44,139 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 2 ViewChangeDone messages 2017-08-28 15:12:44,139 | DEBUG | node.py (1550) | caught_up_for_current_view | Node3 does not have view change quorum for view 0 2017-08-28 15:12:44,139 | DEBUG | node.py (1534) | is_catchup_needed | Node3 is not caught up for the current view 0 2017-08-28 15:12:44,139 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,152 | DEBUG | node.py (1539) | is_catchup_needed | Node3 ordered till last prepared certificate 2017-08-28 15:12:44,152 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-08-28 15:12:44,153 | DEBUG | primary_decider.py ( 131) | send | Node3's elector sending VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0} 2017-08-28 15:12:44,153 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-08-28 15:12:44,153 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:12:44,154 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node1 2017-08-28 15:12:44,163 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,163 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,164 | DEBUG | node.py (2566) | send | Node3 sending message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:12:44,169 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node2 2017-08-28 15:12:44,171 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node2 2017-08-28 15:12:44,171 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node4 2017-08-28 15:12:44,172 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:44,172 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node4 2017-08-28 15:12:44,172 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node1 2017-08-28 15:12:44,172 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}' to Node1 2017-08-28 15:12:44,204 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:44,205 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,216 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node2') 2017-08-28 15:12:44,217 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,217 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,217 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node1') 2017-08-28 15:12:44,217 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,218 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node2') 2017-08-28 15:12:44,218 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node1') 2017-08-28 15:12:44,257 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,257 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node3 got view change quorum (2 >= 2) 2017-08-28 15:12:44,258 | DEBUG | primary_selector.py ( 208) | has_view_change_from_primary | Node3 has not received ViewChangeDone from the next primary Node1 2017-08-28 15:12:44,258 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:12:44,259 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:44,265 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node3 got view change quorum (3 >= 2) 2017-08-28 15:12:44,266 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node3 received ViewChangeDone from primary Node1 2017-08-28 15:12:44,266 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node3 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:12:44,266 | DEBUG | primary_selector.py ( 280) | _startSelection | Node3 starting selection 2017-08-28 15:12:44,266 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 0) 2017-08-28 15:12:44,267 | INFO | node.py ( 503) | start_participating | Node3 started participating 2017-08-28 15:12:44,267 | DEBUG | replica.py ( 406) | primaryName | Node3:0 setting primaryName for view no 0 to: Node1:0 2017-08-28 15:12:44,267 | DEBUG | replica.py (1678) | _gc | Node3:0 cleaning up till (0, 0) 2017-08-28 15:12:44,267 | DEBUG | replica.py (1693) | _gc | Node3:0 found 0 3-phase keys to clean 2017-08-28 15:12:44,275 | DEBUG | replica.py (1695) | _gc | Node3:0 found 0 request keys to clean 2017-08-28 15:12:44,275 | DEBUG | replica.py ( 311) | h | Node3:0 set watermarks as 0 300 2017-08-28 15:12:44,276 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,276 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 0) 2017-08-28 15:12:44,277 | DEBUG | replica.py ( 406) | primaryName | Node3:1 setting primaryName for view no 0 to: Node2:1 2017-08-28 15:12:44,277 | DEBUG | replica.py (1678) | _gc | Node3:1 cleaning up till (0, 0) 2017-08-28 15:12:44,277 | DEBUG | replica.py (1693) | _gc | Node3:1 found 0 3-phase keys to clean 2017-08-28 15:12:44,277 | DEBUG | replica.py (1695) | _gc | Node3:1 found 0 request keys to clean 2017-08-28 15:12:44,277 | DEBUG | replica.py ( 311) | h | Node3:1 set watermarks as 0 300 2017-08-28 15:12:44,277 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node3:1 Setting last ordered for non-master as (0, 0) 2017-08-28 15:12:44,278 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 0) 2017-08-28 15:12:44,278 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,299 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.184 seconds 2017-08-28 15:12:52,086 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:12:52,087 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from Node4 2017-08-28 15:12:52,087 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged Node4 2017-08-28 15:12:52,087 | DEBUG | zstack.py ( 667) | handlePingPong | Node3 got pong from Node4 2017-08-28 15:12:52,093 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:12:52,105 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:12:52,105 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:12:52,106 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:52,107 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:12:52,115 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:52,118 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:52,119 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:12:52,119 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:12:52,124 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:52,124 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node4') 2017-08-28 15:12:52,125 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:52,125 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:52,129 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}}, 'Node4') 2017-08-28 15:12:52,129 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} 2017-08-28 15:12:52,130 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:12:52,130 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,130 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,131 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:12:52,141 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:12:52,144 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,144 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,145 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:12:52,145 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node4 2017-08-28 15:12:52,149 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,149 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,150 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:12:52,150 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} from Node4 2017-08-28 15:12:52,158 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,158 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,158 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:12:52,173 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'po' to Node4 2017-08-28 15:12:52,173 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'po' to Node4 2017-08-28 15:12:52,174 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from {'Node2', 'Node1'} to {'Node2', 'Node4', 'Node1'} 2017-08-28 15:12:52,175 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node4 2017-08-28 15:12:52,178 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from started_hungry to started 2017-08-28 15:12:52,178 | DEBUG | node.py ( 923) | checkInstances | Node3 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node4', 'Node1'} 2017-08-28 15:12:52,182 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node3 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'viewNo': 0}], 'viewNo': 0} to lagged node Node4 2017-08-28 15:12:52,183 | DEBUG | node.py (2566) | send | Node3 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,184 | DEBUG | node.py (2566) | send | Node3 sending message LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,184 | DEBUG | node.py (2566) | send | Node3 sending message LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,184 | DEBUG | node.py (2566) | send | Node3 sending message LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,204 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,204 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0}', b'{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}', b'{"ledgerId":1,"txnSeqNo":9,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"}', b'{"ledgerId":2,"txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-08-28 15:12:52,208 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"}","{\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' 2017-08-28 15:12:52,209 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"}","{\\"ledgerId\\":1,\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"}","{\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"signature":null}' to Node4 2017-08-28 15:12:52,214 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.130 seconds 2017-08-28 15:12:52,478 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:12:52,479 | DEBUG | zstack.py ( 661) | handlePingPong | Node3C got ping from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,481 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'po' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,487 | DEBUG | zstack.py ( 643) | sendPingPong | Node3C ponged b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,552 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:12:52,557 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:52,571 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:52,574 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,575 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,575 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,587 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,588 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,878 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:12:52,878 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-08-28 15:12:52,879 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None}, 'Node4') 2017-08-28 15:12:52,884 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-08-28 15:12:52,884 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} 2017-08-28 15:12:52,887 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,887 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} 2017-08-28 15:12:52,888 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:52,888 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'txnSeqNo': 4, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'Node4') 2017-08-28 15:12:52,888 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} 2017-08-28 15:12:52,896 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,896 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:12:52,896 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,901 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,902 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:12:52,902 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,902 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,909 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:12:52,909 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,910 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,910 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:12:52,911 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:12:52,912 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:52,918 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,919 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:52,923 | DEBUG | node.py ( 889) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} from Node4 2017-08-28 15:12:52,930 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,930 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node4 2017-08-28 15:12:52,931 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,937 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,938 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,938 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,941 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,942 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,944 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,957 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:52,958 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} because it already decided primary which is Node1:0 2017-08-28 15:12:52,958 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-08-28 15:12:52,961 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} because it already decided primary which is Node1:0 2017-08-28 15:12:52,962 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,962 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']) 2017-08-28 15:12:52,971 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:12:52,972 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null}}"],"signature":null}' to Node4 2017-08-28 15:12:52,979 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.103 seconds 2017-08-28 15:12:53,002 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 1 2017-08-28 15:12:53,003 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:12:53,005 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:12:53,007 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:12:53,008 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:53,008 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:53,008 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:12:53,008 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-08-28 15:12:53,380 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:12:53,381 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:12:53,381 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 26, 'viewNo': 1}, 'Node1') 2017-08-28 15:12:53,381 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:12:53,382 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node1 2017-08-28 15:12:53,387 | DEBUG | node.py (1939) | do_view_change_if_possible | Node3 has no quorum for view 1 2017-08-28 15:12:53,388 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:12:53,388 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:53,388 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:53,391 | DEBUG | node.py (1927) | processInstanceChange | Node3 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:12:58,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:58,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:58,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:12:58,369 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:13:03,020 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 4 2017-08-28 15:13:03,021 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:03,031 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:03,035 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:13:03,035 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:03,036 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:03,036 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:03,036 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-08-28 15:13:13,037 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 5 2017-08-28 15:13:13,039 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:13,040 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:13,040 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:13:13,043 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:13,043 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:13,044 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:13,044 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-08-28 15:13:13,376 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:13,377 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:13,378 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:13,378 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:13:20,657 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:13:20,682 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933200605368 2017-08-28 15:13:20,687 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,688 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,688 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,699 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,700 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933200605368 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,704 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:13:20,704 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since not finalised 2017-08-28 15:13:20,705 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:20,712 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node2 2017-08-28 15:13:20,712 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node2 2017-08-28 15:13:20,718 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node4 2017-08-28 15:13:20,718 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node4 2017-08-28 15:13:20,719 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node1 2017-08-28 15:13:20,727 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node1 2017-08-28 15:13:20,730 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:13:20,743 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,746 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,746 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:13:20,747 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,749 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,750 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,750 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}}, 'Node1') 2017-08-28 15:13:20,756 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,757 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,762 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,762 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to 0 replicas 2017-08-28 15:13:20,763 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,774 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,775 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since already forwarded 2017-08-28 15:13:20,786 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:13:20,790 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,795 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,795 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:13:20,796 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,797 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,809 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,809 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since already forwarded 2017-08-28 15:13:20,818 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.163 seconds 2017-08-28 15:13:20,830 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:13:20,838 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]]} 2017-08-28 15:13:20,840 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 1, 'op': 'PREPREPARE', 'ppTime': 1503933200, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]]}, 'Node2') 2017-08-28 15:13:20,840 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]]} 2017-08-28 15:13:20,856 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 1) from Node2:1 at 249497.005453844 2017-08-28 15:13:20,858 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 1) at 249497.007012494 2017-08-28 15:13:20,858 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:13:20,858 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,859 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,863 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)]} with state root None 2017-08-28 15:13:20,863 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 1) 2017-08-28 15:13:20,880 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:13:20,899 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node2 2017-08-28 15:13:20,900 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node2 2017-08-28 15:13:20,903 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node4 2017-08-28 15:13:20,904 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node4 2017-08-28 15:13:20,904 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node1 2017-08-28 15:13:20,906 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":null}' to Node1 2017-08-28 15:13:20,914 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:13:20,918 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]]} 2017-08-28 15:13:20,919 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'discarded': 1, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPREPARE', 'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:20,924 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]]} 2017-08-28 15:13:20,925 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,925 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPARE', 'ppTime': 1503933200, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:20,925 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,945 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 1) from Node1:0 at 249497.093803975 2017-08-28 15:13:20,946 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)]} is b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[', 42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t 2017-08-28 15:13:20,954 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 1) at 249497.103299018 2017-08-28 15:13:20,960 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:13:20,960 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,960 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,961 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)]} with state root b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[' 2017-08-28 15:13:20,966 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 1) 2017-08-28 15:13:20,966 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 1) from Node4:1 2017-08-28 15:13:20,966 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 1) at 249497.115633883 2017-08-28 15:13:20,967 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:13:20,973 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:20,973 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:13:20,973 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:20,975 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:13:20,984 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,984 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPARE', 'ppTime': 1503933200, 'stateRootHash': None, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:20,985 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,994 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:13:21,005 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:13:21,006 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 1) from Node1:1 2017-08-28 15:13:21,006 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,012 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,013 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:13:21,020 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,020 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:13:21,025 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,027 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,027 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'ppSeqNo': 1, 'op': 'PREPARE', 'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd'}, 'Node2') 2017-08-28 15:13:21,031 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,033 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:13:21,042 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']) 2017-08-28 15:13:21,042 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:13:21,048 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' to Node2 2017-08-28 15:13:21,049 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:13:21,049 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']) 2017-08-28 15:13:21,050 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:13:21,055 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' to Node4 2017-08-28 15:13:21,060 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:13:21,060 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']) 2017-08-28 15:13:21,061 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:13:21,068 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":1,\\"instId\\":0,\\"digest\\":\\"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849\\",\\"ppTime\\":1503933200,\\"stateRootHash\\":\\"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd\\"}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"instId\\":1}"],"signature":null}' to Node1 2017-08-28 15:13:21,076 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 1) from Node2:0 2017-08-28 15:13:21,077 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 1) at 249497.225829492 2017-08-28 15:13:21,077 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:13:21,079 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,080 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:13:21,080 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,086 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 1) from Node1:1 2017-08-28 15:13:21,086 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:13:21,086 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,093 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:13:21,094 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,100 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:13:21,100 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,101 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:13:21,101 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node4') 2017-08-28 15:13:21,108 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:13:21,108 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,108 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPARE', 'ppTime': 1503933200, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,109 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,113 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,113 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node4') 2017-08-28 15:13:21,114 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} 2017-08-28 15:13:21,131 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:13:21,136 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 1) from Node4:0 2017-08-28 15:13:21,136 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,139 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,143 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 1) from Node2:1 2017-08-28 15:13:21,144 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:13:21,145 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 1) 2017-08-28 15:13:21,145 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 1) 2017-08-28 15:13:21,145 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:13:21,146 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)]} 2017-08-28 15:13:21,152 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 1) 2017-08-28 15:13:21,152 | DEBUG | replica.py (1540) | _newCheckpointState | Node3:1 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,152 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,157 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:13:21,168 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node2 2017-08-28 15:13:21,169 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node2 2017-08-28 15:13:21,169 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node4 2017-08-28 15:13:21,170 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node4 2017-08-28 15:13:21,177 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node1 2017-08-28 15:13:21,178 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":0}' to Node1 2017-08-28 15:13:21,179 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:13:21,188 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:13:21,189 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,189 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node4') 2017-08-28 15:13:21,195 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,202 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 1) from Node4:0 2017-08-28 15:13:21,212 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:13:21,212 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,216 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:13:21,222 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,226 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:13:21,227 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,227 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,233 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:13:21,233 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} 2017-08-28 15:13:21,241 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 1) from Node2:0 2017-08-28 15:13:21,246 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:13:21,246 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 1) 2017-08-28 15:13:21,247 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 1) 2017-08-28 15:13:21,252 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:13:21,253 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'viewNo': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)]} 2017-08-28 15:13:21,257 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 1) 2017-08-28 15:13:21,257 | DEBUG | replica.py (1540) | _newCheckpointState | Node3:0 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,257 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,258 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:13:21,267 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 1 of 1 requests 2017-08-28 15:13:21,277 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:13:21,277 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to client 2017-08-28 15:13:21,279 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933200,"auditPath":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","reqId":1503933200605368,"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","rootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","dest":"V4SGRU86Z58d6TV7PBUe61","seqNo":10,"identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:21,282 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 1) for ledger 1 range (10, 10) 2017-08-28 15:13:21,282 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:21,302 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.473 seconds 2017-08-28 15:13:23,052 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 6 2017-08-28 15:13:23,053 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:23,053 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:23,053 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:13:23,053 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:23,054 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:23,054 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:23,065 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-08-28 15:13:28,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:28,391 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:28,392 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:28,393 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:13:33,073 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 7 2017-08-28 15:13:33,073 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:33,074 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:33,074 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:13:33,074 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:33,075 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:33,075 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:33,087 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-08-28 15:13:42,988 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 2 2017-08-28 15:13:42,988 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:13:42,989 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:42,991 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 9 to run in 60 seconds 2017-08-28 15:13:42,991 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 1 2017-08-28 15:13:42,992 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:13:42,993 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-08-28 15:13:43,100 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 8 2017-08-28 15:13:43,101 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:43,102 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:43,103 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:13:43,103 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:43,104 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:43,106 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:43,106 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-08-28 15:13:43,346 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action propose_view_change with id 3 2017-08-28 15:13:43,352 | TRACE | node.py (2078) | propose_view_change | The primary is already connected so view change will not be proposed 2017-08-28 15:13:43,403 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:43,411 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:43,411 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:43,413 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:13:53,124 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 10 2017-08-28 15:13:53,124 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:13:53,127 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:53,127 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:13:53,127 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:53,133 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:53,133 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:13:53,134 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-08-28 15:13:58,416 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:58,417 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:58,417 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:58,418 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:14:03,151 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 11 2017-08-28 15:14:03,156 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:03,156 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:03,157 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:03,157 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:03,157 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:03,157 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:03,158 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-08-28 15:14:13,175 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 12 2017-08-28 15:14:13,176 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:13,176 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:13,178 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:13,178 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:13,178 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:13,178 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:13,179 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-08-28 15:14:13,417 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:13,424 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:13,424 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:13,425 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:14:23,198 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 13 2017-08-28 15:14:23,199 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:23,199 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:23,200 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:23,200 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:23,204 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:23,205 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:23,206 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-08-28 15:14:28,429 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:28,430 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:28,430 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:28,431 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:14:33,215 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 14 2017-08-28 15:14:33,217 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:33,217 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:33,217 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:33,218 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:33,218 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:33,218 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:33,219 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-08-28 15:14:43,011 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 9 2017-08-28 15:14:43,011 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:14:43,013 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:43,014 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 16 to run in 60 seconds 2017-08-28 15:14:43,014 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 2 2017-08-28 15:14:43,015 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:14:43,015 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-08-28 15:14:43,232 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 15 2017-08-28 15:14:43,232 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:43,232 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:43,233 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:43,233 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:43,243 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:43,244 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:43,244 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-08-28 15:14:43,434 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:43,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:43,443 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:43,444 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:14:53,251 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 17 2017-08-28 15:14:53,257 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:14:53,258 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:53,258 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:14:53,259 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:53,259 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:53,260 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:14:53,260 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-08-28 15:14:58,436 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:58,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:58,440 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:58,443 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:15:03,277 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 18 2017-08-28 15:15:03,281 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:03,282 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:03,282 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:03,283 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:03,283 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:03,283 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:03,284 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-08-28 15:15:13,295 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 19 2017-08-28 15:15:13,298 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:13,298 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:13,298 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:13,299 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:13,299 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:13,300 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:13,300 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-08-28 15:15:13,438 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:13,438 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:13,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:13,439 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:15:23,314 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 20 2017-08-28 15:15:23,325 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:23,325 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:23,325 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:23,326 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:23,326 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:23,328 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:23,329 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-08-28 15:15:28,453 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:28,454 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:28,454 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:28,461 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:15:33,330 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 21 2017-08-28 15:15:33,337 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:33,338 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:33,338 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:33,338 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:33,338 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:33,339 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:33,345 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-08-28 15:15:43,037 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 16 2017-08-28 15:15:43,040 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:15:43,040 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:43,041 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 23 to run in 60 seconds 2017-08-28 15:15:43,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 3 2017-08-28 15:15:43,045 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:15:43,045 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-08-28 15:15:43,357 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 22 2017-08-28 15:15:43,358 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:43,358 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:43,359 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:43,360 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:43,368 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:43,372 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:43,373 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-08-28 15:15:43,467 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:43,468 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:43,468 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:43,469 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:15:53,385 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 24 2017-08-28 15:15:53,392 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:15:53,392 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:53,396 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:15:53,399 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:53,400 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:53,400 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:15:53,400 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-08-28 15:15:58,495 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:58,496 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:58,496 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:58,500 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:16:03,402 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 25 2017-08-28 15:16:03,402 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:03,402 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:03,403 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:03,403 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:03,416 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:03,416 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:03,416 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-08-28 15:16:13,419 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 26 2017-08-28 15:16:13,419 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:13,421 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:13,421 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:13,427 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:13,427 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:13,428 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:13,434 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-08-28 15:16:13,507 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:13,515 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:13,516 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:13,519 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:16:23,444 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 27 2017-08-28 15:16:23,447 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:23,448 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:23,449 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:23,449 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:23,449 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:23,450 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:23,450 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-08-28 15:16:28,526 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:28,526 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:28,527 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:28,533 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:16:33,471 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 28 2017-08-28 15:16:33,475 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:33,476 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:33,476 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:33,476 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:33,476 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:33,476 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:33,476 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-08-28 15:16:43,042 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 23 2017-08-28 15:16:43,046 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:16:43,046 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:43,066 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 30 to run in 60 seconds 2017-08-28 15:16:43,066 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 4 2017-08-28 15:16:43,066 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:16:43,068 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-08-28 15:16:43,491 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 29 2017-08-28 15:16:43,492 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:43,492 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:43,492 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:43,493 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:43,493 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:43,493 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:43,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-08-28 15:16:43,540 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:43,540 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:43,540 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:43,541 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:16:53,513 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 31 2017-08-28 15:16:53,533 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:16:53,533 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:53,533 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:16:53,534 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:53,534 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:53,534 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:16:53,534 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-08-28 15:16:57,995 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:16:57,996 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:57,997 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:58,001 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:16:58,001 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:16:58,553 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:58,555 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:58,559 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:58,572 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:17:03,541 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 32 2017-08-28 15:17:03,542 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:03,542 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:03,542 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:17:03,542 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:03,542 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:03,542 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:03,542 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-08-28 15:17:07,508 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:07,509 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,511 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,521 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:07,523 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:12,676 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:12,676 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,690 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,707 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:12,708 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:13,559 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:13,560 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:13,560 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:13,562 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:17:13,584 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 33 2017-08-28 15:17:13,585 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:13,595 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:13,600 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:17:13,600 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:13,601 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:13,609 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:13,609 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-08-28 15:17:22,296 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:22,309 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,312 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,326 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:22,340 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:23,614 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 34 2017-08-28 15:17:23,629 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:23,629 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:23,629 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:17:23,631 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:23,637 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:23,638 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:23,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 35 to run in 10 seconds 2017-08-28 15:17:28,065 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:28,084 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:28,084 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:28,086 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:28,093 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:28,594 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:28,595 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:28,598 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:28,599 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:17:33,673 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 35 2017-08-28 15:17:33,673 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:33,673 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:33,676 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8175569048330809 is acceptable. 2017-08-28 15:17:33,676 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:33,688 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:33,688 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:33,688 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-08-28 15:17:37,541 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:37,548 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,548 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,559 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:37,566 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:39,570 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:17:39,592 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933459424488 2017-08-28 15:17:39,593 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,593 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,603 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,604 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,604 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933459424488 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,604 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:39,604 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since not finalised 2017-08-28 15:17:39,605 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:39,632 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node2 2017-08-28 15:17:39,632 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node2 2017-08-28 15:17:39,634 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node4 2017-08-28 15:17:39,634 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node4 2017-08-28 15:17:39,643 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node1 2017-08-28 15:17:39,643 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node1 2017-08-28 15:17:39,660 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:39,662 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,679 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,680 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}}, 'Node1') 2017-08-28 15:17:39,680 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,681 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,690 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,690 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:17:39,690 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,697 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,697 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,697 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:39,710 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,717 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,718 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,719 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to 0 replicas 2017-08-28 15:17:39,724 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,724 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,734 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since already forwarded 2017-08-28 15:17:39,734 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,735 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,747 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'role': '2', 'type': '1', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since already forwarded 2017-08-28 15:17:39,798 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.230 seconds 2017-08-28 15:17:39,849 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:39,850 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]]} 2017-08-28 15:17:39,851 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 2, 'op': 'PREPREPARE', 'ppTime': 1503933459, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]]}, 'Node2') 2017-08-28 15:17:39,851 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]]} 2017-08-28 15:17:39,876 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 2) from Node2:1 at 249756.025618285 2017-08-28 15:17:39,887 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 2) at 249756.036638929 2017-08-28 15:17:39,890 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:17:39,895 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:39,896 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:39,896 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)]} with state root None 2017-08-28 15:17:39,899 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 2) 2017-08-28 15:17:39,912 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:39,922 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]]} 2017-08-28 15:17:39,922 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'discarded': 1, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPREPARE', 'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:39,925 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]]} 2017-08-28 15:17:39,953 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:39,954 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 2) from Node1:0 at 249756.103090027 2017-08-28 15:17:39,955 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)]} is b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t', ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE 2017-08-28 15:17:39,986 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 2) at 249756.13559048 2017-08-28 15:17:39,987 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:17:39,996 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,000 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,000 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)]} with state root b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t' 2017-08-28 15:17:40,010 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 2) 2017-08-28 15:17:40,020 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:40,021 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,021 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPARE', 'ppTime': 1503933459, 'stateRootHash': None, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:40,034 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,034 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,036 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPARE', 'ppTime': 1503933459, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,037 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,060 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node2 2017-08-28 15:17:40,060 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node2 2017-08-28 15:17:40,071 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node4 2017-08-28 15:17:40,072 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node4 2017-08-28 15:17:40,079 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node1 2017-08-28 15:17:40,080 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":null}' to Node1 2017-08-28 15:17:40,092 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:40,099 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 2) from Node1:1 2017-08-28 15:17:40,103 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 2) at 249756.252275341 2017-08-28 15:17:40,106 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:17:40,114 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,115 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:40,115 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,121 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 2) from Node4:1 2017-08-28 15:17:40,122 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,136 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,138 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:40,153 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,161 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'ppSeqNo': 2, 'op': 'PREPARE', 'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR'}, 'Node2') 2017-08-28 15:17:40,171 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,175 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,176 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPARE', 'ppTime': 1503933459, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,176 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,200 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node2 2017-08-28 15:17:40,201 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node2 2017-08-28 15:17:40,202 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node4 2017-08-28 15:17:40,209 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node4 2017-08-28 15:17:40,217 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node1 2017-08-28 15:17:40,218 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","viewNo":0,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ppTime":1503933459,"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR"}' to Node1 2017-08-28 15:17:40,230 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:40,240 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 2) from Node2:0 2017-08-28 15:17:40,240 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 2) at 249756.389359427 2017-08-28 15:17:40,241 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:17:40,247 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,248 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:40,248 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,254 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 2) from Node4:0 2017-08-28 15:17:40,258 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,261 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,290 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node2 2017-08-28 15:17:40,291 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node2 2017-08-28 15:17:40,304 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node4 2017-08-28 15:17:40,305 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node4 2017-08-28 15:17:40,315 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node1 2017-08-28 15:17:40,316 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":1}' to Node1 2017-08-28 15:17:40,317 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:40,339 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:40,341 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,342 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:40,350 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,353 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,353 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:40,353 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,359 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,359 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:40,364 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:40,387 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node2 2017-08-28 15:17:40,388 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node2 2017-08-28 15:17:40,388 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node4 2017-08-28 15:17:40,393 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node4 2017-08-28 15:17:40,395 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node1 2017-08-28 15:17:40,405 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":2,"instId":0}' to Node1 2017-08-28 15:17:40,407 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 2) from Node2:1 2017-08-28 15:17:40,414 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:40,414 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,415 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 2) from Node1:1 2017-08-28 15:17:40,429 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:17:40,429 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 2) 2017-08-28 15:17:40,430 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 2) 2017-08-28 15:17:40,430 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:17:40,441 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)]} 2017-08-28 15:17:40,441 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 2) 2017-08-28 15:17:40,442 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,449 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:40,451 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:40,471 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,472 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:40,472 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,473 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,473 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:40,479 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,486 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,486 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:40,489 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:40,523 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:17:40,524 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 2) from Node2:0 2017-08-28 15:17:40,524 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:40,526 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,532 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 2) from Node1:0 2017-08-28 15:17:40,532 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:17:40,533 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 2) 2017-08-28 15:17:40,533 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 2) 2017-08-28 15:17:40,533 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:17:40,543 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'viewNo': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)]} 2017-08-28 15:17:40,544 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 2) 2017-08-28 15:17:40,544 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,555 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:40,559 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 2 of 1 requests 2017-08-28 15:17:40,581 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:40,592 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to client 2017-08-28 15:17:40,593 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933459,"auditPath":["A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","reqId":1503933459424488,"verkey":"~RmCt3RtDvKDfQBSKxo4qvy","rootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":11}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:40,604 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 2) for ledger 1 range (11, 11) 2017-08-28 15:17:40,612 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:40,641 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.806 seconds 2017-08-28 15:17:42,745 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:42,747 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,749 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,771 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:42,775 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:43,070 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 30 2017-08-28 15:17:43,070 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:17:43,071 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:43,071 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 37 to run in 60 seconds 2017-08-28 15:17:43,071 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 5 2017-08-28 15:17:43,073 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:17:43,073 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-08-28 15:17:43,606 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:43,615 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:43,617 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:43,618 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:17:43,694 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 36 2017-08-28 15:17:43,706 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:43,706 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:43,715 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8738318825604503 is acceptable. 2017-08-28 15:17:43,720 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:43,720 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:43,720 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:43,721 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-08-28 15:17:46,205 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:17:46,206 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933466149129 2017-08-28 15:17:46,208 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,216 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,216 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,222 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,222 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933466149129 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,223 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:46,236 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since not finalised 2017-08-28 15:17:46,236 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:46,244 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node2 2017-08-28 15:17:46,252 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node2 2017-08-28 15:17:46,258 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node4 2017-08-28 15:17:46,258 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node4 2017-08-28 15:17:46,265 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node1 2017-08-28 15:17:46,265 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node1 2017-08-28 15:17:46,267 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:46,287 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,293 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,298 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:17:46,298 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,306 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,307 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,309 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}}, 'Node1') 2017-08-28 15:17:46,309 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,319 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,320 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,328 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to 0 replicas 2017-08-28 15:17:46,329 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,341 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,341 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since already forwarded 2017-08-28 15:17:46,356 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:46,367 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,379 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,380 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:46,380 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,388 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,388 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,392 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'role': '2', 'type': '1', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since already forwarded 2017-08-28 15:17:46,430 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.242 seconds 2017-08-28 15:17:46,461 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:46,480 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]]} 2017-08-28 15:17:46,480 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 3, 'op': 'PREPREPARE', 'ppTime': 1503933466, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]]}, 'Node2') 2017-08-28 15:17:46,480 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]]} 2017-08-28 15:17:46,506 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 3) from Node2:1 at 249762.644118774 2017-08-28 15:17:46,515 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 3) at 249762.663710468 2017-08-28 15:17:46,517 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:17:46,519 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,520 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,520 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)]} with state root None 2017-08-28 15:17:46,520 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 3) 2017-08-28 15:17:46,530 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:46,542 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]]} 2017-08-28 15:17:46,542 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'discarded': 1, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPREPARE', 'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:46,542 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]]} 2017-08-28 15:17:46,557 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:46,560 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 3) from Node1:0 at 249762.708704784 2017-08-28 15:17:46,575 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)]} is b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`', 32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb 2017-08-28 15:17:46,578 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 3) at 249762.726691106 2017-08-28 15:17:46,591 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:17:46,592 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,592 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,592 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)]} with state root b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`' 2017-08-28 15:17:46,600 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 3) 2017-08-28 15:17:46,613 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:46,614 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,615 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'ppSeqNo': 3, 'op': 'PREPARE', 'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4'}, 'Node2') 2017-08-28 15:17:46,616 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,616 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,630 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPARE', 'ppTime': 1503933466, 'stateRootHash': None, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:46,632 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,632 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,633 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPARE', 'ppTime': 1503933466, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,633 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,649 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node2 2017-08-28 15:17:46,656 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node2 2017-08-28 15:17:46,656 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node4 2017-08-28 15:17:46,664 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node4 2017-08-28 15:17:46,664 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node1 2017-08-28 15:17:46,664 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":null}' to Node1 2017-08-28 15:17:46,679 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:46,686 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 3) from Node2:0 2017-08-28 15:17:46,686 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 3) at 249762.835670541 2017-08-28 15:17:46,687 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:17:46,688 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:46,691 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:46,699 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,699 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 3) from Node1:1 2017-08-28 15:17:46,706 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 3) at 249762.854863219 2017-08-28 15:17:46,706 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:17:46,706 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,720 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:46,720 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,729 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 3) from Node4:1 2017-08-28 15:17:46,730 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,730 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,738 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:46,755 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,756 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPARE', 'ppTime': 1503933466, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,756 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,777 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node2 2017-08-28 15:17:46,785 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node2 2017-08-28 15:17:46,786 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node4 2017-08-28 15:17:46,787 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node4 2017-08-28 15:17:46,797 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node1 2017-08-28 15:17:46,798 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","viewNo":0,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ppTime":1503933466,"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4"}' to Node1 2017-08-28 15:17:46,808 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:46,821 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:46,821 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 3) from Node4:0 2017-08-28 15:17:46,822 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,822 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,824 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:46,825 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,825 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:46,825 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,864 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:17:46,865 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:17:46,873 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:17:46,874 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node2 2017-08-28 15:17:46,875 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:17:46,880 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:17:46,880 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:17:46,891 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node4 2017-08-28 15:17:46,892 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:17:46,892 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:17:46,893 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:17:46,903 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node1 2017-08-28 15:17:46,914 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 3) from Node2:1 2017-08-28 15:17:46,916 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:46,924 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:46,925 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:46,940 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,940 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:46,940 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,941 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:17:46,952 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node4') 2017-08-28 15:17:46,952 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:17:46,955 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:46,963 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:46,963 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:46,970 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,970 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:46,970 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:46,971 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:46,981 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:46,982 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:46,998 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 3) from Node4:0 2017-08-28 15:17:47,013 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:47,013 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,014 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 3) from Node2:0 2017-08-28 15:17:47,014 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:17:47,024 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 3) 2017-08-28 15:17:47,024 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 3) 2017-08-28 15:17:47,025 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:17:47,033 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'viewNo': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)]} 2017-08-28 15:17:47,033 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 3) 2017-08-28 15:17:47,034 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,042 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 3) from Node1:1 2017-08-28 15:17:47,042 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:17:47,051 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 3) 2017-08-28 15:17:47,051 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 3) 2017-08-28 15:17:47,052 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:17:47,052 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)]} 2017-08-28 15:17:47,053 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 3) 2017-08-28 15:17:47,058 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,059 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:47,071 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:47,080 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:47,087 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:47,091 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:47,112 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 3 of 1 requests 2017-08-28 15:17:47,136 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:47,143 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to client 2017-08-28 15:17:47,144 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933466,"auditPath":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","reqId":1503933466149129,"verkey":"~2a4KepmnVVAPGo5gNA3P3p","rootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":12}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:47,153 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 3) for ledger 1 range (12, 12) 2017-08-28 15:17:47,154 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:17:47,178 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:47,183 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:47,253 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.793 seconds 2017-08-28 15:17:51,717 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:17:51,730 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933471684615 2017-08-28 15:17:51,735 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,743 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,744 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,752 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,759 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933471684615 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,762 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:51,771 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since not finalised 2017-08-28 15:17:51,778 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:51,789 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node2 2017-08-28 15:17:51,789 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node2 2017-08-28 15:17:51,797 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node4 2017-08-28 15:17:51,801 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node4 2017-08-28 15:17:51,801 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node1 2017-08-28 15:17:51,802 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node1 2017-08-28 15:17:51,822 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:51,830 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,834 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,834 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}}, 'Node1') 2017-08-28 15:17:51,839 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,840 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,848 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,848 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:51,848 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,856 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,857 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,863 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:17:51,864 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,864 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,875 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,876 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to 0 replicas 2017-08-28 15:17:51,876 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,888 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,892 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since already forwarded 2017-08-28 15:17:51,893 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,896 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,904 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '6XiqM9DwYDgFnmyQQBFr72', 'role': '100', 'type': '1', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since already forwarded 2017-08-28 15:17:51,963 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.272 seconds 2017-08-28 15:17:51,983 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:51,984 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]]} 2017-08-28 15:17:51,985 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 4, 'op': 'PREPREPARE', 'ppTime': 1503933471, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]]}, 'Node2') 2017-08-28 15:17:51,992 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]]} 2017-08-28 15:17:52,007 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 4) from Node2:1 at 249768.1559092 2017-08-28 15:17:52,020 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 4) at 249768.1687291 2017-08-28 15:17:52,020 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:17:52,024 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,024 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,024 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)]} with state root None 2017-08-28 15:17:52,026 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 4) 2017-08-28 15:17:52,036 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:52,047 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]]} 2017-08-28 15:17:52,050 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'discarded': 1, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPREPARE', 'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,051 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]]} 2017-08-28 15:17:52,065 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:52,078 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 4) from Node1:0 at 249768.227304952 2017-08-28 15:17:52,080 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)]} is b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3', 8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg 2017-08-28 15:17:52,082 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 4) at 249768.231034325 2017-08-28 15:17:52,097 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:17:52,097 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,098 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,098 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)]} with state root b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3' 2017-08-28 15:17:52,110 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 4) 2017-08-28 15:17:52,131 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node2 2017-08-28 15:17:52,132 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node2 2017-08-28 15:17:52,133 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node4 2017-08-28 15:17:52,142 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node4 2017-08-28 15:17:52,143 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node1 2017-08-28 15:17:52,151 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":null}' to Node1 2017-08-28 15:17:52,153 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:52,170 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:52,171 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,171 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPARE', 'ppTime': 1503933471, 'stateRootHash': None, 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,171 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,196 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node2 2017-08-28 15:17:52,197 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node2 2017-08-28 15:17:52,202 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node4 2017-08-28 15:17:52,203 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node4 2017-08-28 15:17:52,212 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node1 2017-08-28 15:17:52,212 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","viewNo":0,"op":"PREPARE","ppSeqNo":4,"instId":0,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ppTime":1503933471,"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV"}' to Node1 2017-08-28 15:17:52,213 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 4) from Node1:1 2017-08-28 15:17:52,228 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 4) at 249768.376762268 2017-08-28 15:17:52,228 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:17:52,228 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,230 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:52,231 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,252 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:52,252 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,258 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPARE', 'ppTime': 1503933471, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,258 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,258 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,269 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'ppSeqNo': 4, 'op': 'PREPARE', 'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV'}, 'Node2') 2017-08-28 15:17:52,269 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,292 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:52,299 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 4) from Node2:0 2017-08-28 15:17:52,303 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 4) at 249768.452434256 2017-08-28 15:17:52,304 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:17:52,304 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,315 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:52,315 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,322 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 4) from Node4:1 2017-08-28 15:17:52,322 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,331 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,339 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:52,349 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,349 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,350 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,350 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPARE', 'ppTime': 1503933471, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,350 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,373 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node2 2017-08-28 15:17:52,381 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node2 2017-08-28 15:17:52,381 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node4 2017-08-28 15:17:52,382 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node4 2017-08-28 15:17:52,391 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node1 2017-08-28 15:17:52,392 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":1}' to Node1 2017-08-28 15:17:52,392 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:52,401 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:52,414 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:52,422 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 4) from Node4:0 2017-08-28 15:17:52,423 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,423 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,435 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:52,441 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,441 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:52,448 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,448 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,463 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:52,475 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,485 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node2 2017-08-28 15:17:52,492 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node2 2017-08-28 15:17:52,500 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node4 2017-08-28 15:17:52,500 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node4 2017-08-28 15:17:52,503 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node1 2017-08-28 15:17:52,503 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":4,"instId":0}' to Node1 2017-08-28 15:17:52,505 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 4) from Node1:1 2017-08-28 15:17:52,505 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:52,505 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,505 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 4) from Node4:1 2017-08-28 15:17:52,505 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:17:52,505 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 4) 2017-08-28 15:17:52,506 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 4) 2017-08-28 15:17:52,506 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:17:52,506 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)]} 2017-08-28 15:17:52,506 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 4) 2017-08-28 15:17:52,506 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,546 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:52,552 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,552 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:52,552 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:52,553 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,553 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:52,564 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,581 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:17:52,595 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 4) from Node4:0 2017-08-28 15:17:52,602 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:52,602 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,603 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 4, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:52,614 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:52,623 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,633 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:52,633 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,653 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 4) from Node2:0 2017-08-28 15:17:52,663 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:17:52,664 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 4) 2017-08-28 15:17:52,664 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 4) 2017-08-28 15:17:52,669 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:17:52,670 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'viewNo': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'ppSeqNo': 4, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)]} 2017-08-28 15:17:52,670 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 4) 2017-08-28 15:17:52,675 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,677 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:52,693 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,694 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:52,703 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:52,732 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 4 of 1 requests 2017-08-28 15:17:52,735 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:52,736 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to client 2017-08-28 15:17:52,736 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933471,"auditPath":["4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","reqId":1503933471684615,"verkey":"~YHDYsnaN59TzUsLqvPYdRr","rootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":13}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:52,749 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 4) for ledger 1 range (13, 13) 2017-08-28 15:17:52,749 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:52,749 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:52,797 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.823 seconds 2017-08-28 15:17:53,749 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 38 2017-08-28 15:17:53,758 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:17:53,762 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:53,763 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8931957598312256 is acceptable. 2017-08-28 15:17:53,767 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:53,767 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:53,771 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:17:53,772 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-08-28 15:17:57,625 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:57,626 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil': INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:17:57,636 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 26, 'viewNo': 1}, b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil') 2017-08-28 15:17:57,636 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:17:57,637 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:57,640 | DEBUG | node.py (1939) | do_view_change_if_possible | Node3 has no quorum for view 1 2017-08-28 15:17:57,640 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8931957598312256 is acceptable. 2017-08-28 15:17:57,640 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:57,657 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:57,657 | DEBUG | node.py (1927) | processInstanceChange | Node3 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:17:58,095 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:58,111 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:58,113 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:58,127 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:58,130 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:58,610 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:58,614 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:58,615 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:58,627 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:17:58,914 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:17:58,945 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933478716086 2017-08-28 15:17:58,956 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,956 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,965 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,965 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,966 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933478716086 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,978 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:58,983 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since not finalised 2017-08-28 15:17:58,984 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:58,992 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node2 2017-08-28 15:17:58,993 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node2 2017-08-28 15:17:59,006 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node4 2017-08-28 15:17:59,007 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node4 2017-08-28 15:17:59,007 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node1 2017-08-28 15:17:59,016 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node1 2017-08-28 15:17:59,018 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:59,040 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,040 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,041 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}}, 'Node1') 2017-08-28 15:17:59,042 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,055 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,056 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,056 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:59,061 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,062 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,070 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,070 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:17:59,070 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,080 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,081 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,088 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to 0 replicas 2017-08-28 15:17:59,089 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,095 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,095 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since already forwarded 2017-08-28 15:17:59,096 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,096 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,107 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'role': '101', 'type': '1', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since already forwarded 2017-08-28 15:17:59,151 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.239 seconds 2017-08-28 15:17:59,202 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:59,208 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]]} 2017-08-28 15:17:59,222 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'discarded': 1, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPREPARE', 'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,222 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]]} 2017-08-28 15:17:59,222 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]]} 2017-08-28 15:17:59,223 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 5, 'op': 'PREPREPARE', 'ppTime': 1503933479, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]]}, 'Node2') 2017-08-28 15:17:59,233 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]]} 2017-08-28 15:17:59,252 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 5) from Node1:0 at 249775.401018224 2017-08-28 15:17:59,253 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)]} is b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6', CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA 2017-08-28 15:17:59,268 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 5) at 249775.417186444 2017-08-28 15:17:59,278 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:17:59,279 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,283 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,284 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)]} with state root b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6' 2017-08-28 15:17:59,291 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 5) 2017-08-28 15:17:59,292 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 5) from Node2:1 at 249775.440822409 2017-08-28 15:17:59,292 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 5) at 249775.441409163 2017-08-28 15:17:59,305 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:17:59,306 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,306 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,310 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)]} with state root None 2017-08-28 15:17:59,312 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 5) 2017-08-28 15:17:59,343 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:59,359 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:59,361 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:17:59,373 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,373 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPARE', 'ppTime': 1503933479, 'stateRootHash': None, 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,374 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,385 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:17:59,391 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']) 2017-08-28 15:17:59,403 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:17:59,408 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' to Node2 2017-08-28 15:17:59,421 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:17:59,421 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']) 2017-08-28 15:17:59,422 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:17:59,432 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' to Node4 2017-08-28 15:17:59,443 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:17:59,451 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']) 2017-08-28 15:17:59,457 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:17:59,457 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":0,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":\\"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":5,\\"instId\\":1,\\"digest\\":\\"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210\\",\\"ppTime\\":1503933479,\\"stateRootHash\\":null}"],"signature":null}' to Node1 2017-08-28 15:17:59,477 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 5) from Node1:1 2017-08-28 15:17:59,478 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 5) at 249775.626845595 2017-08-28 15:17:59,480 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:17:59,480 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,481 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:59,490 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,497 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:59,502 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,502 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPARE', 'ppTime': 1503933479, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,511 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,514 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,515 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'ppSeqNo': 5, 'op': 'PREPARE', 'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1'}, 'Node2') 2017-08-28 15:17:59,515 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,525 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,525 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPARE', 'ppTime': 1503933479, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,533 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,556 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:59,557 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 5) from Node2:0 2017-08-28 15:17:59,559 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 5) at 249775.70863065 2017-08-28 15:17:59,563 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:17:59,564 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,564 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:59,564 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,575 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 5) from Node4:0 2017-08-28 15:17:59,578 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,578 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,579 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 5) from Node4:1 2017-08-28 15:17:59,593 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,593 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,618 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node2 2017-08-28 15:17:59,619 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node2 2017-08-28 15:17:59,628 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node4 2017-08-28 15:17:59,635 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node4 2017-08-28 15:17:59,637 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node1 2017-08-28 15:17:59,637 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":1}' to Node1 2017-08-28 15:17:59,639 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:17:59,656 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:17:59,656 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,670 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:59,670 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,670 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,675 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:59,676 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,676 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:17:59,676 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node4') 2017-08-28 15:17:59,680 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:17:59,686 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,686 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 5, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:59,686 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,696 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,696 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 5, 'viewNo': 0}, 'Node4') 2017-08-28 15:17:59,705 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} 2017-08-28 15:17:59,723 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node2 2017-08-28 15:17:59,724 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node2 2017-08-28 15:17:59,735 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node4 2017-08-28 15:17:59,736 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node4 2017-08-28 15:17:59,744 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node1 2017-08-28 15:17:59,744 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":0,"ppSeqNo":5,"instId":0}' to Node1 2017-08-28 15:17:59,753 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 5) from Node4:0 2017-08-28 15:17:59,764 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:59,764 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,764 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 5) from Node2:1 2017-08-28 15:17:59,775 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:17:59,775 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,781 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 5) from Node1:1 2017-08-28 15:17:59,781 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:17:59,781 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 5) 2017-08-28 15:17:59,782 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 5) 2017-08-28 15:17:59,782 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:17:59,782 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)]} 2017-08-28 15:17:59,791 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 5) 2017-08-28 15:17:59,792 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,793 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 5, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:59,813 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:17:59,815 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,815 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:17:59,823 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,826 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,827 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:17:59,827 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} 2017-08-28 15:17:59,846 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:17:59,860 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 5) from Node2:0 2017-08-28 15:17:59,860 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:17:59,860 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 5) 2017-08-28 15:17:59,867 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 5) 2017-08-28 15:17:59,868 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:17:59,868 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'viewNo': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'ppSeqNo': 5, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)]} 2017-08-28 15:17:59,880 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 5) 2017-08-28 15:17:59,880 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,881 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 5, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:17:59,898 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 5 of 1 requests 2017-08-28 15:17:59,916 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:59,917 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to client 2017-08-28 15:17:59,931 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933479,"auditPath":["HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","reqId":1503933478716086,"verkey":"~RTJS62PRHurqHJnbnT1Hgi","rootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":14}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:59,939 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 5) for ledger 1 range (14, 14) 2017-08-28 15:17:59,944 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:59,994 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.793 seconds 2017-08-28 15:18:03,782 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 39 2017-08-28 15:18:03,794 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:03,794 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:03,794 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node3 master throughput ratio 0.8950231700533599 is acceptable. 2017-08-28 15:18:03,795 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:03,795 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:03,796 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:03,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-08-28 15:18:04,219 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:18:04,235 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933484172953 2017-08-28 15:18:04,244 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,245 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,250 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,253 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,254 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933484172953 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,260 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:04,261 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '1', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'dest': '5qhSKMkpqui6taGLTecwqf'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since not finalised 2017-08-28 15:18:04,261 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:04,272 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node2 2017-08-28 15:18:04,281 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node2 2017-08-28 15:18:04,282 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node4 2017-08-28 15:18:04,282 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node4 2017-08-28 15:18:04,288 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node1 2017-08-28 15:18:04,289 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"1","verkey":"~UWK9xwiKsqMQSBmymYpR5v","dest":"5qhSKMkpqui6taGLTecwqf"},"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node1 2017-08-28 15:18:04,304 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:04,316 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,321 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,322 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}}, 'Node1') 2017-08-28 15:18:04,322 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,335 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,335 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,336 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:04,336 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,337 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,352 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,352 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:18:04,352 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,353 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,362 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,362 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to 0 replicas 2017-08-28 15:18:04,366 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,373 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,373 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since already forwarded 2017-08-28 15:18:04,374 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,384 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,384 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v', 'type': '1'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since already forwarded 2017-08-28 15:18:04,443 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.228 seconds 2017-08-28 15:18:04,517 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:04,520 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]]} 2017-08-28 15:18:04,523 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppSeqNo': 6, 'op': 'PREPREPARE', 'ppTime': 1503933484, 'stateRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]]}, 'Node2') 2017-08-28 15:18:04,523 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]]} 2017-08-28 15:18:04,528 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]]} 2017-08-28 15:18:04,528 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 1, 'viewNo': 0, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'discarded': 1, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPREPARE', 'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:04,528 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]]} 2017-08-28 15:18:04,542 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(0, 6) from Node1:0 at 249780.691176247 2017-08-28 15:18:04,560 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)]} is b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>', Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn 2017-08-28 15:18:04,561 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(0, 6) at 249780.710516094 2017-08-28 15:18:04,578 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:18:04,579 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,580 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,583 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)]} with state root b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>' 2017-08-28 15:18:04,589 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(0, 6) 2017-08-28 15:18:04,590 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:1 received PRE-PREPARE(0, 6) from Node2:1 at 249780.738877461 2017-08-28 15:18:04,598 | DEBUG | replica.py ( 934) | doPrepare | Node3:1 Sending PREPARE(0, 6) at 249780.746987009 2017-08-28 15:18:04,598 | DEBUG | replica.py (2133) | send | Node3:1 sending Prepare 2017-08-28 15:18:04,602 | TRACE | replica.py (2134) | send | Node3:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,603 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,603 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 1, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)]} with state root None 2017-08-28 15:18:04,608 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:1 processed incoming PRE-PREPARE(0, 6) 2017-08-28 15:18:04,636 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:04,636 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:04,637 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:04,649 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,651 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPARE', 'ppTime': 1503933484, 'stateRootHash': None, 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:04,651 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,662 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,662 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'ppSeqNo': 6, 'op': 'PREPARE', 'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz'}, 'Node2') 2017-08-28 15:18:04,663 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,666 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:04,684 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']) 2017-08-28 15:18:04,686 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:18:04,687 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' to Node2 2017-08-28 15:18:04,700 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:04,701 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']) 2017-08-28 15:18:04,701 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:18:04,714 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' to Node4 2017-08-28 15:18:04,715 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:04,722 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', b'{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']) 2017-08-28 15:18:04,722 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' 2017-08-28 15:18:04,727 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"txnRootHash\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":0,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":\\"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz\\"}","{\\"txnRootHash\\":null,\\"viewNo\\":0,\\"op\\":\\"PREPARE\\",\\"ppSeqNo\\":6,\\"instId\\":1,\\"digest\\":\\"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337\\",\\"ppTime\\":1503933484,\\"stateRootHash\\":null}"],"signature":null}' to Node1 2017-08-28 15:18:04,738 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 6) from Node2:0 2017-08-28 15:18:04,738 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(0, 6) at 249780.887488214 2017-08-28 15:18:04,739 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:18:04,751 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:04,751 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:18:04,751 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,757 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 6) from Node1:1 2017-08-28 15:18:04,758 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(0, 6) at 249780.906798186 2017-08-28 15:18:04,767 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:18:04,767 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,767 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:18:04,772 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,773 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:04,793 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,794 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPARE', 'ppTime': 1503933484, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,794 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,818 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:04,829 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:04,834 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(0, 6) from Node4:1 2017-08-28 15:18:04,835 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,835 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,848 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:04,855 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,862 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPARE', 'ppTime': 1503933484, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,862 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,881 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:04,882 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']) 2017-08-28 15:18:04,884 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:18:04,892 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' to Node2 2017-08-28 15:18:04,892 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:04,902 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']) 2017-08-28 15:18:04,903 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:18:04,908 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' to Node4 2017-08-28 15:18:04,911 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:04,912 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', b'{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']) 2017-08-28 15:18:04,912 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:18:04,919 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":0,\\"ppSeqNo\\":6,\\"instId\\":1}"],"signature":null}' to Node1 2017-08-28 15:18:04,927 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(0, 6) from Node4:0 2017-08-28 15:18:04,931 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,931 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,949 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:04,949 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,954 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2') 2017-08-28 15:18:04,954 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,955 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:18:04,959 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:04,960 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:18:04,960 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:04,961 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 6, 'viewNo': 0}, 'Node4') 2017-08-28 15:18:04,961 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:04,961 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,973 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 6, 'viewNo': 0}, 'Node4') 2017-08-28 15:18:04,974 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,974 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,987 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1') 2017-08-28 15:18:04,988 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} 2017-08-28 15:18:04,998 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 6) from Node4:0 2017-08-28 15:18:05,012 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} commits where f is 1 2017-08-28 15:18:05,012 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,013 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 6) from Node2:1 2017-08-28 15:18:05,026 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} commits where f is 1 2017-08-28 15:18:05,027 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,031 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(0, 6) from Node4:1 2017-08-28 15:18:05,031 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:18:05,032 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (0, 6) 2017-08-28 15:18:05,032 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (0, 6) 2017-08-28 15:18:05,032 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:18:05,032 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'stateRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)]} 2017-08-28 15:18:05,032 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (0, 6) 2017-08-28 15:18:05,044 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,044 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 6, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:18:05,046 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:05,064 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:05,070 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2') 2017-08-28 15:18:05,070 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:05,071 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:05,079 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1') 2017-08-28 15:18:05,079 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} 2017-08-28 15:18:05,093 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:18:05,108 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(0, 6) from Node2:0 2017-08-28 15:18:05,113 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:18:05,113 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (0, 6) 2017-08-28 15:18:05,114 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (0, 6) 2017-08-28 15:18:05,114 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:18:05,132 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 1, 'instId': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'ppSeqNo': 6, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)]} 2017-08-28 15:18:05,132 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (0, 6) 2017-08-28 15:18:05,132 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,133 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 6, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message 2017-08-28 15:18:05,152 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 0 6 of 1 requests 2017-08-28 15:18:05,174 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:18:05,179 | DEBUG | node.py (2344) | sendReplyToClient | Node3 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to client 2017-08-28 15:18:05,179 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REPLY","result":{"txnTime":1503933484,"auditPath":["ijqq5QRv5GQwp9riPYGkR5Xkz9QxMds39Wc21R4V9Pt","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","reqId":1503933484172953,"verkey":"~UWK9xwiKsqMQSBmymYpR5v","type":"1","rootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","dest":"5qhSKMkpqui6taGLTecwqf","seqNo":15,"identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:05,192 | DEBUG | node.py (2263) | executeBatch | Node3 storing 3PC key (0, 6) for ledger 1 range (15, 15) 2017-08-28 15:18:05,192 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:05,246 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.730 seconds 2017-08-28 15:18:07,158 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:07,168 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7': INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:18:07,176 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 26, 'viewNo': 1}, b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7') 2017-08-28 15:18:07,186 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:18:07,187 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,195 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 1 from 0 2017-08-28 15:18:07,199 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action _check_view_change_completed with id 41 to run in 60 seconds 2017-08-28 15:18:07,204 | DEBUG | replica.py ( 453) | on_view_change_start | Node3:0 setting last prepared for master to (0, 6) 2017-08-28 15:18:07,204 | DEBUG | node.py (2127) | startViewChange | Node3 resetting monitor stats after view change 2017-08-28 15:18:07,204 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:18:07,205 | DEBUG | replica.py ( 406) | primaryName | Node3:0 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,212 | DEBUG | replica.py ( 406) | primaryName | Node3:1 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,249 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node3 changed to view 1, will start catchup now 2017-08-28 15:18:07,285 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,285 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,285 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,286 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,287 | DEBUG | node.py (2173) | start_catchup | Node3 reverted 0 batches before starting catch up 2017-08-28 15:18:07,287 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,288 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 0 2017-08-28 15:18:07,289 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,289 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 0 2017-08-28 15:18:07,289 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:07,299 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 0 2017-08-28 15:18:07,308 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,313 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 0 2017-08-28 15:18:07,313 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:07,320 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:07,320 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:07,320 | DEBUG | node.py (1927) | processInstanceChange | Node3 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:18:07,327 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,363 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,364 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,371 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:18:07,372 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,377 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,377 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,383 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:18:07,392 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,392 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,392 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,397 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:18:07,409 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:07,410 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:07,419 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:07,420 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:07,420 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,422 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:07,447 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,460 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,460 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:07,460 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,467 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,472 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,473 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,473 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,474 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:07,484 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,484 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,492 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:07,493 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,493 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,493 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:07,494 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,502 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,503 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:07,508 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,509 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,509 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:07,515 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,516 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,528 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,529 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,548 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,549 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,549 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,562 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,571 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:18:07,572 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:07,582 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:18:07,592 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,595 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:18:07,598 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:07,611 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node4 2017-08-28 15:18:07,614 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,617 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:18:07,632 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:07,639 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:18:07,654 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 4 messages through listener 2017-08-28 15:18:07,664 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,664 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,665 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,678 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,681 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,682 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,691 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node1') 2017-08-28 15:18:07,692 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,692 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,700 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node1') 2017-08-28 15:18:07,700 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,700 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:07,700 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:07,710 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:07,711 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,717 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node4') 2017-08-28 15:18:07,717 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,717 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,729 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node4') 2017-08-28 15:18:07,729 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,732 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,740 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:07,741 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:18:07,749 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,749 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node2') 2017-08-28 15:18:07,750 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,750 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,750 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node2') 2017-08-28 15:18:07,768 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:18:07,768 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node1 2017-08-28 15:18:07,776 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,776 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,777 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node1 2017-08-28 15:18:07,778 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,783 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,784 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node4 2017-08-28 15:18:07,796 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,796 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,796 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 0 is latest 2017-08-28 15:18:07,803 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} that it does not need catchup 2017-08-28 15:18:07,804 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:18:07,804 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,814 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,814 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,814 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,820 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:18:07,820 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:18:07,820 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:18:07,821 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,821 | DEBUG | node.py ( 923) | checkInstances | Node3 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node4', 'Node1'} 2017-08-28 15:18:07,821 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,822 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 2 2017-08-28 15:18:07,822 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,822 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 2 2017-08-28 15:18:07,822 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:07,822 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 2 2017-08-28 15:18:07,822 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,822 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 2 2017-08-28 15:18:07,823 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:18:07,823 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:18:07,845 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node4 2017-08-28 15:18:07,855 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,859 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,860 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node2 2017-08-28 15:18:07,861 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,869 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,869 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:18:07,870 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node2 2017-08-28 15:18:07,880 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,880 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,880 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:18:07,890 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,896 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,905 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,905 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:18:07,906 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,911 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,911 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,919 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:18:07,920 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,927 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:07,927 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:07,928 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:18:07,928 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:07,944 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:07,946 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:07,959 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:07,960 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,967 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:07,968 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,968 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:07,968 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,976 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,976 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:07,983 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:07,989 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:07,990 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:07,990 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:07,999 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,001 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:08,002 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,002 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,015 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:08,015 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,015 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,016 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:08,016 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,022 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,027 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:08,027 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,028 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,028 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:08,028 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,040 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,049 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,049 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,056 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,073 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,074 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,084 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,095 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:18:08,096 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:08,107 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:18:08,109 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,110 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:18:08,116 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:08,125 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node4 2017-08-28 15:18:08,126 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,126 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:18:08,135 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:18:08,140 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:18:08,161 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:08,168 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,178 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:08,179 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,184 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,184 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:18:08,185 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,185 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,199 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:18:08,199 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,200 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,205 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,206 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:18:08,222 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,222 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:18:08,222 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,228 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,229 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:18:08,229 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,239 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:08,245 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:08,246 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:08,254 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,255 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:18:08,259 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,268 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,268 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:18:08,268 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:18:08,278 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:18:08,279 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,288 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,289 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:18:08,290 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,297 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,298 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:18:08,312 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,312 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,312 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:18:08,313 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} that it does not need catchup 2017-08-28 15:18:08,320 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,320 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,320 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,327 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,328 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,328 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:18:08,338 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:18:08,338 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:18:08,339 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,349 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:18:08,350 | DEBUG | upgrader.py ( 208) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya processing config ledger for any upgrades 2017-08-28 15:18:08,350 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,360 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 1 2017-08-28 15:18:08,361 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,362 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 1 2017-08-28 15:18:08,380 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:08,380 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 1 2017-08-28 15:18:08,384 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,392 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 1 2017-08-28 15:18:08,392 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:18:08,393 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-08-28 15:18:08,400 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:18:08,408 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:18:08,416 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,416 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,428 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:18:08,429 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,430 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,430 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:18:08,437 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:18:08,438 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,438 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,439 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:18:08,465 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,465 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:08,466 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:08,468 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:18:08,469 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,474 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:08,482 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:08,483 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:18:08,483 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,495 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:18:08,495 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:18:08,498 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:18:08,509 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:08,521 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,521 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:08,521 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,522 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,523 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:08,532 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,532 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,533 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:18:08,541 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,542 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,542 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,548 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,549 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,549 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:08,549 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,558 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,561 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:18:08,561 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,568 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:08,568 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:08,568 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:18:08,575 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,575 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:08,581 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,582 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,582 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:18:08,594 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:18:08,595 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,596 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,612 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,613 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,614 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,625 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,637 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,640 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:18:08,640 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:18:08,653 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node2 2017-08-28 15:18:08,654 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,659 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:18:08,663 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:18:08,664 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node4 2017-08-28 15:18:08,675 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,678 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:18:08,678 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:18:08,684 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node1 2017-08-28 15:18:08,702 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:08,708 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,722 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:18:08,723 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,729 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,729 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node2') 2017-08-28 15:18:08,729 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,730 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,730 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node2') 2017-08-28 15:18:08,743 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,744 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,744 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,745 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:18:08,755 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,756 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node1') 2017-08-28 15:18:08,756 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,761 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,761 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node1') 2017-08-28 15:18:08,767 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,768 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:08,778 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:18:08,778 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:18:08,779 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,783 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node4') 2017-08-28 15:18:08,784 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,791 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,794 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node4') 2017-08-28 15:18:08,794 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:18:08,795 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node2 2017-08-28 15:18:08,805 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,805 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,806 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node2 2017-08-28 15:18:08,817 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,817 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,818 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node1 2017-08-28 15:18:08,828 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,828 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,828 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:18:08,829 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} that it does not need catchup 2017-08-28 15:18:08,835 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,835 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,836 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,836 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,837 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,845 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:18:08,845 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:18:08,845 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:18:08,846 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,857 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:18:08,857 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,858 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:18:08,859 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:18:08,860 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 3 ViewChangeDone messages 2017-08-28 15:18:08,868 | DEBUG | node.py (1550) | caught_up_for_current_view | Node3 does not have view change quorum for view 1 2017-08-28 15:18:08,869 | DEBUG | node.py (1534) | is_catchup_needed | Node3 is not caught up for the current view 1 2017-08-28 15:18:08,869 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,875 | DEBUG | node.py (1539) | is_catchup_needed | Node3 ordered till last prepared certificate 2017-08-28 15:18:08,876 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-08-28 15:18:08,877 | DEBUG | primary_decider.py ( 131) | send | Node3's elector sending VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 1} 2017-08-28 15:18:08,883 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 2 ViewChangeDone messages 2017-08-28 15:18:08,884 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:08,884 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node1 2017-08-28 15:18:08,896 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,896 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,896 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node4 2017-08-28 15:18:08,909 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,909 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,914 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:18:08,914 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node4 2017-08-28 15:18:08,921 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,921 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,921 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:18:08,932 | DEBUG | node.py (2566) | send | Node3 sending message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:08,943 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node2 2017-08-28 15:18:08,944 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node2 2017-08-28 15:18:08,948 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node4 2017-08-28 15:18:08,948 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node4 2017-08-28 15:18:08,949 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node1 2017-08-28 15:18:08,957 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}' to Node1 2017-08-28 15:18:08,977 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:08,978 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:08,979 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node4') 2017-08-28 15:18:08,980 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:08,993 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:08,993 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node2') 2017-08-28 15:18:08,994 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,000 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,000 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node1') 2017-08-28 15:18:09,000 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,012 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node4') 2017-08-28 15:18:09,012 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node2') 2017-08-28 15:18:09,020 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node1') 2017-08-28 15:18:09,034 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,034 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-08-28 15:18:09,035 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:09,035 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,044 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node3 got view change quorum (3 >= 3) 2017-08-28 15:18:09,045 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node3 received ViewChangeDone from primary Node2 2017-08-28 15:18:09,045 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node3 found acceptable primary Node2 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:18:09,052 | DEBUG | primary_selector.py ( 280) | _startSelection | Node3 starting selection 2017-08-28 15:18:09,052 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node2:0 for instance 0 (view 1) 2017-08-28 15:18:09,053 | INFO | node.py ( 503) | start_participating | Node3 started participating 2017-08-28 15:18:09,062 | DEBUG | replica.py ( 406) | primaryName | Node3:0 setting primaryName for view no 1 to: Node2:0 2017-08-28 15:18:09,062 | DEBUG | replica.py (1678) | _gc | Node3:0 cleaning up till (0, 6) 2017-08-28 15:18:09,062 | DEBUG | replica.py (1693) | _gc | Node3:0 found 6 3-phase keys to clean 2017-08-28 15:18:09,063 | DEBUG | replica.py (1695) | _gc | Node3:0 found 6 request keys to clean 2017-08-28 15:18:09,069 | DEBUG | replica.py ( 311) | h | Node3:0 set watermarks as 0 300 2017-08-28 15:18:09,070 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,070 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node3:1 for instance 1 (view 1) 2017-08-28 15:18:09,078 | DEBUG | replica.py ( 406) | primaryName | Node3:1 setting primaryName for view no 1 to: Node3:1 2017-08-28 15:18:09,078 | DEBUG | replica.py (1678) | _gc | Node3:1 cleaning up till (0, 6) 2017-08-28 15:18:09,079 | DEBUG | replica.py (1693) | _gc | Node3:1 found 6 3-phase keys to clean 2017-08-28 15:18:09,091 | DEBUG | replica.py (1695) | _gc | Node3:1 found 6 request keys to clean 2017-08-28 15:18:09,091 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) from previous checkpoints 2017-08-28 15:18:09,092 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) from previous checkpoints 2017-08-28 15:18:09,092 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) from previous checkpoints 2017-08-28 15:18:09,092 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) from previous checkpoints 2017-08-28 15:18:09,097 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) from previous checkpoints 2017-08-28 15:18:09,097 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) from previous checkpoints 2017-08-28 15:18:09,103 | DEBUG | replica.py ( 311) | h | Node3:1 set watermarks as 0 300 2017-08-28 15:18:09,104 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node3:1 Setting last ordered for non-master as (0, 6) 2017-08-28 15:18:09,104 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (1, 0) 2017-08-28 15:18:09,104 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,113 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-08-28 15:18:09,113 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} because it already decided primary which is Node2:0 2017-08-28 15:18:09,130 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.974 seconds 2017-08-28 15:18:12,738 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:12,745 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,746 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,761 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:12,768 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:13,612 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:13,613 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:13,613 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:13,613 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:18:13,806 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 40 2017-08-28 15:18:13,816 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:13,817 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:13,817 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:13,817 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:13,818 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:13,830 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:13,830 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-08-28 15:18:15,671 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:18:15,702 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933495609306 2017-08-28 15:18:15,702 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,703 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,704 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,704 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,716 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933495609306 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,717 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:15,719 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since not finalised 2017-08-28 15:18:15,728 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:15,739 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node2 2017-08-28 15:18:15,742 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node2 2017-08-28 15:18:15,751 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node4 2017-08-28 15:18:15,762 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node4 2017-08-28 15:18:15,771 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node1 2017-08-28 15:18:15,774 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node1 2017-08-28 15:18:15,788 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:15,789 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,795 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,800 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}}, 'Node1') 2017-08-28 15:18:15,807 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,818 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,818 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,824 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:15,829 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,836 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,847 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,848 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:18:15,848 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,863 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,867 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,868 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) to 0 replicas 2017-08-28 15:18:15,869 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,887 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,888 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since already forwarded 2017-08-28 15:18:15,889 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,898 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,907 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since already forwarded 2017-08-28 15:18:15,923 | DEBUG | replica.py ( 665) | create3PCBatch | Node3:1 creating batch 1 for ledger 0 with state root None 2017-08-28 15:18:15,938 | DEBUG | replica.py ( 698) | create3PCBatch | Node3:1 created a PRE-PREPARE with 1 requests for ledger 0 2017-08-28 15:18:15,938 | DEBUG | replica.py (2133) | send | Node3:1 sending PrePrepare 2017-08-28 15:18:15,939 | TRACE | replica.py (2134) | send | Node3:1 sending PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} 2017-08-28 15:18:15,939 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} with state root None 2017-08-28 15:18:15,959 | DEBUG | node.py (2566) | send | Node3 sending message PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:15,984 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node2 2017-08-28 15:18:15,985 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node2 2017-08-28 15:18:15,987 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node4 2017-08-28 15:18:15,994 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node4 2017-08-28 15:18:15,995 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node1 2017-08-28 15:18:15,999 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"ledgerId":0,"instId":1,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":null}' to Node1 2017-08-28 15:18:16,027 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.355 seconds 2017-08-28 15:18:16,062 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:16,078 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]]} 2017-08-28 15:18:16,078 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'ppSeqNo': 1, 'op': 'PREPREPARE', 'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]]}, 'Node2') 2017-08-28 15:18:16,079 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]]} 2017-08-28 15:18:16,090 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(1, 1) from Node2:0 at 249792.239218995 2017-08-28 15:18:16,113 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:18:16,114 | WARNING | replica.py ( 655) | processReqDuringBatch | Node3:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}, will reject 2017-08-28 15:18:16,118 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(1, 1) at 249792.26708118 2017-08-28 15:18:16,118 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:18:16,119 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,126 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,127 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:16,140 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(1, 1) 2017-08-28 15:18:16,141 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:16,141 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,151 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPARE', 'ppTime': 1503933495, 'stateRootHash': None, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,155 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,178 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REJECT","reqId":1503933495609306,"reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:16,182 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:16,182 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 1) from Node1:1 2017-08-28 15:18:16,183 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,187 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,196 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:16,197 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,207 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'ppSeqNo': 1, 'op': 'PREPARE', 'ppTime': 1503933495, 'stateRootHash': None}, 'Node2') 2017-08-28 15:18:16,210 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,210 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,210 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPARE', 'ppTime': 1503933495, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,211 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,235 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:18:16,251 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:18:16,256 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:18:16,256 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:18:16,269 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:18:16,270 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ppTime":1503933495,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:18:16,273 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 1) from Node2:1 2017-08-28 15:18:16,290 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(1, 1) at 249792.43955352 2017-08-28 15:18:16,291 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:18:16,291 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,296 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:16,296 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,296 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 1) from Node4:1 2017-08-28 15:18:16,301 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,301 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,302 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:16,313 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,313 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPARE', 'ppTime': 1503933495, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,324 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,325 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,328 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node2') 2017-08-28 15:18:16,329 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,345 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:16,357 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 1) from Node4:0 2017-08-28 15:18:16,358 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(1, 1) at 249792.506715663 2017-08-28 15:18:16,358 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:18:16,358 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,360 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:16,362 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,368 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 1) from Node2:1 2017-08-28 15:18:16,368 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:16,368 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,380 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:16,388 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,398 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPARE', 'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,402 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,410 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node2 2017-08-28 15:18:16,418 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node2 2017-08-28 15:18:16,419 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node4 2017-08-28 15:18:16,425 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node4 2017-08-28 15:18:16,426 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node1 2017-08-28 15:18:16,426 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":1}' to Node1 2017-08-28 15:18:16,436 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:16,437 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 1) from Node1:0 2017-08-28 15:18:16,443 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,444 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,460 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:16,461 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,467 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node4') 2017-08-28 15:18:16,467 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,468 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,469 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node1') 2017-08-28 15:18:16,470 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:16,484 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node2 2017-08-28 15:18:16,486 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node2 2017-08-28 15:18:16,495 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node4 2017-08-28 15:18:16,496 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node4 2017-08-28 15:18:16,501 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node1 2017-08-28 15:18:16,501 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":1,"instId":0}' to Node1 2017-08-28 15:18:16,503 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 1) from Node4:1 2017-08-28 15:18:16,521 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:18:16,521 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (1, 1) 2017-08-28 15:18:16,521 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (1, 1) 2017-08-28 15:18:16,523 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:18:16,524 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'stateRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)]} 2017-08-28 15:18:16,531 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (1, 1) 2017-08-28 15:18:16,532 | DEBUG | replica.py (1540) | _newCheckpointState | Node3:1 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,532 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,532 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:18:16,542 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:16,548 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,548 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node2') 2017-08-28 15:18:16,558 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,571 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:18:16,578 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 1) from Node2:0 2017-08-28 15:18:16,579 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:16,579 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,593 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:16,594 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,602 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node4') 2017-08-28 15:18:16,602 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,615 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,616 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node1') 2017-08-28 15:18:16,616 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:16,651 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 1) from Node4:0 2017-08-28 15:18:16,652 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:18:16,652 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (1, 1) 2017-08-28 15:18:16,652 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (1, 1) 2017-08-28 15:18:16,659 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:18:16,660 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'ppSeqNo': 1, 'reqIdr': []} 2017-08-28 15:18:16,660 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (1, 1) 2017-08-28 15:18:16,660 | DEBUG | replica.py (1540) | _newCheckpointState | Node3:0 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,668 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,669 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:18:16,678 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 1 1 of 0 requests 2017-08-28 15:18:16,679 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:16,707 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:16,727 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.666 seconds 2017-08-28 15:18:22,303 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:22,311 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:22,315 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:22,317 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:22,328 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:23,230 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:18:23,247 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933503177919 2017-08-28 15:18:23,253 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,265 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,266 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,274 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,275 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933503177919 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,275 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:23,287 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since not finalised 2017-08-28 15:18:23,293 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,305 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node2 2017-08-28 15:18:23,306 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node2 2017-08-28 15:18:23,319 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node4 2017-08-28 15:18:23,320 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node4 2017-08-28 15:18:23,331 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node1 2017-08-28 15:18:23,332 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node1 2017-08-28 15:18:23,345 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:23,356 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,363 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,364 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}}, 'Node1') 2017-08-28 15:18:23,371 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,373 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,386 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,387 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:23,395 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,402 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,402 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,411 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2') 2017-08-28 15:18:23,412 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,422 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,423 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,435 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) to 0 replicas 2017-08-28 15:18:23,435 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,445 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,445 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since already forwarded 2017-08-28 15:18:23,459 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,463 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,468 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since already forwarded 2017-08-28 15:18:23,470 | DEBUG | replica.py ( 665) | create3PCBatch | Node3:1 creating batch 2 for ledger 0 with state root None 2017-08-28 15:18:23,492 | DEBUG | replica.py ( 698) | create3PCBatch | Node3:1 created a PRE-PREPARE with 1 requests for ledger 0 2017-08-28 15:18:23,492 | DEBUG | replica.py (2133) | send | Node3:1 sending PrePrepare 2017-08-28 15:18:23,492 | TRACE | replica.py (2134) | send | Node3:1 sending PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} 2017-08-28 15:18:23,495 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} with state root None 2017-08-28 15:18:23,509 | DEBUG | node.py (2566) | send | Node3 sending message PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:23,547 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node2 2017-08-28 15:18:23,548 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node2 2017-08-28 15:18:23,548 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node4 2017-08-28 15:18:23,548 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node4 2017-08-28 15:18:23,556 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node1 2017-08-28 15:18:23,557 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"ledgerId":0,"instId":1,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":null}' to Node1 2017-08-28 15:18:23,588 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.360 seconds 2017-08-28 15:18:23,625 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:23,639 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]]} 2017-08-28 15:18:23,643 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'ppSeqNo': 2, 'op': 'PREPREPARE', 'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]]}, 'Node2') 2017-08-28 15:18:23,644 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]]} 2017-08-28 15:18:23,660 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(1, 2) from Node2:0 at 249799.809180295 2017-08-28 15:18:23,662 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:18:23,676 | WARNING | replica.py ( 655) | processReqDuringBatch | Node3:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}, will reject 2017-08-28 15:18:23,677 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(1, 2) at 249799.825688244 2017-08-28 15:18:23,687 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:18:23,687 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,691 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,692 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:23,693 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(1, 2) 2017-08-28 15:18:23,710 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REJECT","reqId":1503933503177919,"reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,733 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:23,743 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:23,745 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,745 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'ppSeqNo': 2, 'op': 'PREPARE', 'ppTime': 1503933503, 'stateRootHash': None}, 'Node2') 2017-08-28 15:18:23,746 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,751 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,752 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPARE', 'ppTime': 1503933503, 'stateRootHash': None, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:23,752 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,772 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:18:23,780 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:18:23,781 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:18:23,781 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:18:23,796 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:18:23,796 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":2,"instId":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ppTime":1503933503,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:18:23,800 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 2) from Node2:1 2017-08-28 15:18:23,808 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,811 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,812 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 2) from Node1:1 2017-08-28 15:18:23,812 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(1, 2) at 249799.961377679 2017-08-28 15:18:23,835 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:18:23,836 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:23,836 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:23,837 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,853 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:23,854 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,879 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPARE', 'ppTime': 1503933503, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:23,880 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,886 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,886 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPARE', 'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:23,887 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,902 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 42 2017-08-28 15:18:23,903 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:23,903 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:23,903 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:23,907 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:23,907 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:23,911 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:23,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 43 to run in 10 seconds 2017-08-28 15:18:23,926 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:23,930 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 2) from Node1:0 2017-08-28 15:18:23,931 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(1, 2) at 249800.079981669 2017-08-28 15:18:23,931 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:18:23,934 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:23,935 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:23,940 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,946 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 2) from Node4:1 2017-08-28 15:18:23,946 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,951 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,953 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:18:23,959 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:23,968 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node1') 2017-08-28 15:18:23,971 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:23,972 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:23,972 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node2') 2017-08-28 15:18:23,984 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:23,999 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node2 2017-08-28 15:18:24,000 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node2 2017-08-28 15:18:24,000 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node4 2017-08-28 15:18:24,000 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node4 2017-08-28 15:18:24,007 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node1 2017-08-28 15:18:24,007 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":1}' to Node1 2017-08-28 15:18:24,018 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:18:24,026 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 2) from Node1:1 2017-08-28 15:18:24,026 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:24,026 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,033 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 2) from Node2:1 2017-08-28 15:18:24,033 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:18:24,033 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (1, 2) 2017-08-28 15:18:24,034 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (1, 2) 2017-08-28 15:18:24,034 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:18:24,043 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'stateRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)]} 2017-08-28 15:18:24,044 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (1, 2) 2017-08-28 15:18:24,052 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,053 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:24,061 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:24,064 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPARE', 'ppTime': 1503933503, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,069 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:24,083 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node2 2017-08-28 15:18:24,084 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node2 2017-08-28 15:18:24,094 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node4 2017-08-28 15:18:24,095 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node4 2017-08-28 15:18:24,096 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node1 2017-08-28 15:18:24,096 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"COMMIT","viewNo":1,"ppSeqNo":2,"instId":0}' to Node1 2017-08-28 15:18:24,106 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:18:24,117 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 2) from Node4:0 2017-08-28 15:18:24,117 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:24,117 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:24,118 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:18:24,130 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:24,130 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node4') 2017-08-28 15:18:24,130 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} 2017-08-28 15:18:24,131 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,131 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node2') 2017-08-28 15:18:24,144 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,147 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,147 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node1') 2017-08-28 15:18:24,148 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,165 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 2) from Node2:0 2017-08-28 15:18:24,180 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:18:24,181 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,181 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 2) from Node1:0 2017-08-28 15:18:24,181 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:18:24,182 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (1, 2) 2017-08-28 15:18:24,187 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (1, 2) 2017-08-28 15:18:24,188 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:18:24,188 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'ppSeqNo': 2, 'reqIdr': []} 2017-08-28 15:18:24,188 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (1, 2) 2017-08-28 15:18:24,198 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,199 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:18:24,208 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:24,209 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,211 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node4') 2017-08-28 15:18:24,217 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} 2017-08-28 15:18:24,233 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 1 2 of 0 requests 2017-08-28 15:18:24,247 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:24,249 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:24,249 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:18:24,297 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.672 seconds 2017-08-28 15:18:28,091 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:28,092 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,093 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,094 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:28,102 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:28,616 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:28,624 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:28,627 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:28,628 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:18:33,922 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 43 2017-08-28 15:18:33,934 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:33,934 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:33,935 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:33,935 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:33,946 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:33,946 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:33,947 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 44 to run in 10 seconds 2017-08-28 15:18:37,705 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:37,714 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,722 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,734 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:37,737 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:42,782 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:42,789 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,790 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,804 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:42,804 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:43,098 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 37 2017-08-28 15:18:43,101 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:18:43,101 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:43,111 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 45 to run in 60 seconds 2017-08-28 15:18:43,112 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 6 2017-08-28 15:18:43,112 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:18:43,112 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-08-28 15:18:43,640 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:43,662 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:43,662 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:43,663 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:18:43,958 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 44 2017-08-28 15:18:43,958 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:43,959 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:43,959 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:43,963 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:43,967 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:43,978 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:43,978 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-08-28 15:18:52,322 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:52,327 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,333 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,348 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:52,355 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:54,006 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 46 2017-08-28 15:18:54,007 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:18:54,007 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:54,008 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:18:54,008 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:54,009 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:54,012 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:18:54,012 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-08-28 15:18:58,030 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:18:58,036 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,051 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,053 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:58,071 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:58,646 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:58,663 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:58,664 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:58,668 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:19:04,021 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 47 2017-08-28 15:19:04,031 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:04,031 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:04,032 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:04,032 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:04,032 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:04,032 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:04,032 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-08-28 15:19:07,218 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action _check_view_change_completed with id 41 2017-08-28 15:19:07,235 | DEBUG | node.py ( 993) | _check_view_change_completed | Node3 running the scheduled check for view change completion 2017-08-28 15:19:07,236 | DEBUG | node.py ( 995) | _check_view_change_completed | Node3 already completion view change 2017-08-28 15:19:07,634 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:07,652 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,652 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,657 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:07,660 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:12,819 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:12,820 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,820 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,835 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:12,836 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:13,668 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:13,669 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:13,669 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:13,673 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:19:14,050 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 48 2017-08-28 15:19:14,050 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:14,058 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:14,058 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:14,059 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:14,068 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:14,068 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:14,069 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-08-28 15:19:22,358 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:22,372 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,372 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,395 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:22,396 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:24,070 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 49 2017-08-28 15:19:24,078 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:24,078 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:24,083 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:24,089 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:24,090 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:24,090 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:24,090 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-08-28 15:19:28,044 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:28,045 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,045 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,073 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:28,074 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:28,685 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:28,690 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:28,691 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:28,695 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:19:34,110 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 50 2017-08-28 15:19:34,113 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:34,113 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:34,113 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:34,114 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:34,114 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:34,114 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:34,114 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 51 to run in 10 seconds 2017-08-28 15:19:37,528 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:37,531 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,531 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,553 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:37,554 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:42,806 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:42,819 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,828 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,836 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:42,836 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:43,126 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 45 2017-08-28 15:19:43,126 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:19:43,127 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:43,144 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 52 to run in 60 seconds 2017-08-28 15:19:43,146 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 7 2017-08-28 15:19:43,146 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:19:43,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-08-28 15:19:43,696 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:43,700 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:43,701 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:43,702 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:19:44,115 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 51 2017-08-28 15:19:44,116 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:44,116 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:44,117 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:44,117 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:44,117 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:44,139 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:44,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-08-28 15:19:52,378 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:52,381 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,392 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,393 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:52,407 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:54,161 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 53 2017-08-28 15:19:54,162 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:19:54,163 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:54,164 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:19:54,164 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:54,176 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:54,177 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:19:54,177 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-08-28 15:19:58,202 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:19:58,203 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,210 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,217 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:58,231 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:58,708 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:58,728 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:58,729 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:58,730 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:20:04,188 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 54 2017-08-28 15:20:04,190 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:04,200 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:04,200 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:04,200 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:04,201 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:04,204 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:04,207 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-08-28 15:20:07,505 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:07,505 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,516 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,517 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:07,517 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:12,814 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:12,815 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,815 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,834 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:12,844 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:13,720 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:13,722 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:13,730 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:13,731 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:20:14,231 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 55 2017-08-28 15:20:14,240 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:14,241 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:14,242 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:14,247 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:14,254 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:14,254 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:14,255 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-08-28 15:20:22,395 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:22,396 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,396 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,415 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:22,424 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:24,269 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 56 2017-08-28 15:20:24,271 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:24,271 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:24,291 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:24,295 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:24,296 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:24,296 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:24,296 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-08-28 15:20:28,172 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:28,177 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,185 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,190 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:28,195 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:28,740 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:28,741 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:28,741 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:28,742 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:20:34,304 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 57 2017-08-28 15:20:34,305 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:34,309 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:34,310 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:34,310 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:34,311 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:34,311 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:34,313 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-08-28 15:20:37,611 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:37,616 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,617 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,639 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:37,640 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:42,853 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:42,854 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,854 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,872 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:42,872 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:43,138 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 52 2017-08-28 15:20:43,144 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:20:43,144 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:43,144 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds 2017-08-28 15:20:43,168 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 8 2017-08-28 15:20:43,169 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:20:43,169 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-08-28 15:20:43,749 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:43,755 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:43,760 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:43,760 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:20:44,319 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 58 2017-08-28 15:20:44,320 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:44,323 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:44,336 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:44,336 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:44,336 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:44,337 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:44,337 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-08-28 15:20:52,426 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:52,434 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,434 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,444 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:52,445 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:54,354 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 60 2017-08-28 15:20:54,354 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:20:54,355 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:54,359 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:20:54,363 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:54,364 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:54,364 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:20:54,364 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 61 to run in 10 seconds 2017-08-28 15:20:58,222 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:20:58,223 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,223 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,246 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:58,247 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:58,765 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:58,770 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:58,779 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:58,780 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:21:04,391 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 61 2017-08-28 15:21:04,394 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:04,394 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:04,394 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:04,395 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:04,396 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:04,399 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:04,399 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-08-28 15:21:07,647 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:07,651 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,653 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,655 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:07,668 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:12,885 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:12,886 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,886 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,905 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:12,916 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:13,770 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:13,784 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:13,784 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:13,790 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:21:14,403 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 62 2017-08-28 15:21:14,405 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:14,405 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:14,406 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:14,409 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:14,414 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:14,414 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:14,414 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-08-28 15:21:22,438 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:22,445 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,446 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,451 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:22,466 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:24,414 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 63 2017-08-28 15:21:24,434 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:24,434 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:24,434 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:24,434 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:24,435 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:24,435 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:24,438 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-08-28 15:21:28,219 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:28,221 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,222 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,235 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:28,235 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:28,783 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:28,787 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:28,787 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:28,793 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:21:34,448 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 64 2017-08-28 15:21:34,451 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:34,454 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:34,455 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:34,463 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:34,463 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:34,464 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:34,464 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-08-28 15:21:37,644 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:37,655 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,661 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,682 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:37,683 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:42,907 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:42,908 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,908 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,922 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:42,922 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:43,164 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 59 2017-08-28 15:21:43,169 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:21:43,169 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:43,170 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 66 to run in 60 seconds 2017-08-28 15:21:43,171 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 9 2017-08-28 15:21:43,172 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:21:43,172 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-08-28 15:21:43,813 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:43,814 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:43,816 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:43,820 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:21:44,482 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 65 2017-08-28 15:21:44,486 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:44,499 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:44,500 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:44,500 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:44,502 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:44,508 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:44,510 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-08-28 15:21:52,518 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:52,523 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,524 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,538 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:52,539 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:54,514 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 67 2017-08-28 15:21:54,530 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:21:54,530 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:54,531 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:21:54,532 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:54,534 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:54,535 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:21:54,535 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-08-28 15:21:58,132 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:21:58,145 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,145 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,147 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:58,147 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:58,820 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:58,839 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:58,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:58,840 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:22:04,543 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 68 2017-08-28 15:22:04,545 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:04,545 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:04,545 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:04,545 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:04,546 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:04,548 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:04,549 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-08-28 15:22:07,701 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:07,714 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,715 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,726 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:07,727 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:12,917 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:12,918 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,922 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,937 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:12,939 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:13,843 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:13,846 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:13,846 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:13,848 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:22:14,559 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 69 2017-08-28 15:22:14,576 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:14,576 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:14,577 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:14,577 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:14,579 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:14,580 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:14,580 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-08-28 15:22:22,582 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:22,587 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,588 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,624 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:22,625 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:24,595 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 70 2017-08-28 15:22:24,596 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:24,596 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:24,596 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:24,598 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:24,599 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:24,601 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:24,601 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-08-28 15:22:28,289 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:28,296 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,314 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,318 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:28,327 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:28,877 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:28,879 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:28,881 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:28,882 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:22:34,604 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 71 2017-08-28 15:22:34,605 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:34,605 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:34,605 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:34,609 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:34,611 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:34,615 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:34,618 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-08-28 15:22:37,786 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:37,786 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,789 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,799 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:37,819 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:42,965 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:42,966 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,973 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,988 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:42,988 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:43,176 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 66 2017-08-28 15:22:43,188 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:22:43,188 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:43,188 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 73 to run in 60 seconds 2017-08-28 15:22:43,188 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 10 2017-08-28 15:22:43,189 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:22:43,191 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-08-28 15:22:43,879 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:43,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:43,888 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:43,889 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:22:44,632 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 72 2017-08-28 15:22:44,633 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:44,640 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:44,640 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:44,641 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:44,646 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:44,647 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:44,650 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-08-28 15:22:52,566 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:52,567 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,572 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,581 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:52,592 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:54,672 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 74 2017-08-28 15:22:54,675 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:22:54,683 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:54,695 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:22:54,702 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:54,702 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:54,708 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:22:54,708 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-08-28 15:22:58,280 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:22:58,287 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,290 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,292 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:58,311 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:58,889 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:58,894 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:58,894 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:58,898 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:23:04,722 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 75 2017-08-28 15:23:04,724 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:04,724 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:04,724 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:04,724 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:04,725 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:04,726 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:04,728 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-08-28 15:23:07,760 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:07,774 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,774 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,788 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:07,792 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:12,979 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:12,992 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:13,001 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:13,003 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:13,012 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:13,913 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:13,919 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:13,923 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:13,926 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:23:14,742 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 76 2017-08-28 15:23:14,746 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:14,753 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:14,754 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:14,754 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:14,771 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:14,773 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:14,773 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 77 to run in 10 seconds 2017-08-28 15:23:22,586 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:22,602 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,602 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,613 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:22,614 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:24,804 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 77 2017-08-28 15:23:24,805 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:24,805 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:24,805 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:24,805 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:24,806 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:24,806 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:24,806 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-08-28 15:23:28,268 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:28,280 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,280 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,292 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:28,293 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:28,938 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:28,939 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:28,942 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:28,943 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:23:34,829 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 78 2017-08-28 15:23:34,839 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:34,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:34,840 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:34,843 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:34,848 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:34,848 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:34,848 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-08-28 15:23:37,833 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:37,834 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,846 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,852 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:37,852 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:42,990 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:42,990 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:42,990 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:42,992 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:43,006 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:43,219 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 73 2017-08-28 15:23:43,220 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:23:43,220 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:43,221 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 80 to run in 60 seconds 2017-08-28 15:23:43,227 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 11 2017-08-28 15:23:43,228 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:23:43,228 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-08-28 15:23:43,946 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:43,959 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:43,960 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:43,961 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:23:44,865 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 79 2017-08-28 15:23:44,876 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:44,876 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:44,877 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:44,877 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:44,877 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:44,878 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:44,879 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-08-28 15:23:52,570 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:52,576 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,576 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,580 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:52,597 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:54,881 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 81 2017-08-28 15:23:54,882 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:23:54,882 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:54,883 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:23:54,885 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:54,887 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:54,888 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:23:54,888 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-08-28 15:23:58,285 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:23:58,285 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,291 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,293 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:58,304 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:58,946 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:58,947 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:58,947 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:58,963 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:24:04,911 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 82 2017-08-28 15:24:04,915 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:04,920 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:04,920 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:04,920 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:04,921 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:04,924 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:04,924 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-08-28 15:24:07,732 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:07,733 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,740 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,741 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:07,759 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:12,973 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:12,973 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:12,979 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:12,981 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:12,994 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:13,979 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:13,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:13,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:13,995 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:24:14,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 83 2017-08-28 15:24:14,928 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:14,928 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:14,932 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:14,932 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:14,932 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:14,935 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:14,936 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-08-28 15:24:22,609 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:22,610 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,610 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,631 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:22,646 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:24,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 84 2017-08-28 15:24:24,949 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:24,949 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:24,951 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:24,952 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:24,952 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:24,966 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:24,966 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 85 to run in 10 seconds 2017-08-28 15:24:28,298 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:28,303 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,313 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,316 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:28,329 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:28,984 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:28,988 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:28,988 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:29,002 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:24:34,980 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 85 2017-08-28 15:24:34,981 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:34,981 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:34,981 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:34,989 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:34,999 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:35,000 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:35,000 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-08-28 15:24:37,761 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:37,775 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,775 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,789 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:37,789 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:43,010 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:43,010 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,029 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,032 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:43,046 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:43,226 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 80 2017-08-28 15:24:43,234 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:24:43,235 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:43,236 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 87 to run in 60 seconds 2017-08-28 15:24:43,240 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 12 2017-08-28 15:24:43,242 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:24:43,248 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-08-28 15:24:43,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:43,996 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:43,997 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:43,997 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:24:45,012 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 86 2017-08-28 15:24:45,012 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:45,012 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:45,012 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:45,018 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:45,022 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:45,023 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:45,023 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-08-28 15:24:52,619 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:52,621 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,622 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,631 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:52,639 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:55,046 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 88 2017-08-28 15:24:55,047 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:24:55,047 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:55,049 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:24:55,049 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:55,049 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:55,052 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:24:55,053 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-08-28 15:24:58,352 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:24:58,353 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,354 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,369 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:58,374 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:59,010 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:59,010 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:59,011 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:59,013 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:25:05,061 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 89 2017-08-28 15:25:05,061 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:05,064 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:05,067 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:05,070 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:05,082 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:05,082 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:25:05,083 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-08-28 15:25:08,018 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:08,020 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:08,024 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:08,051 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:08,052 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:13,054 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:13,062 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,062 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,069 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:13,076 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:14,014 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:14,021 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:14,021 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:14,022 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:25:15,097 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 90 2017-08-28 15:25:15,099 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:15,100 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:15,101 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:15,103 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:15,107 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:15,108 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:25:15,109 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-08-28 15:25:22,619 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:22,622 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,627 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,637 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:22,646 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:25,137 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 91 2017-08-28 15:25:25,139 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:25,140 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:25,140 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:25,153 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:25,155 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:25,156 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:25:25,156 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-08-28 15:25:28,323 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:28,328 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,328 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,338 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:28,338 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:29,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:29,042 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:29,042 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:29,042 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:25:35,175 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 92 2017-08-28 15:25:35,179 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:35,180 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:35,180 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:35,180 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:35,181 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:35,181 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:25:35,195 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 93 to run in 10 seconds 2017-08-28 15:25:37,961 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:37,962 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,963 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,990 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:37,990 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:43,102 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:43,108 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,108 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,124 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:43,125 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:43,241 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 87 2017-08-28 15:25:43,243 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node3 checking its request amount 2017-08-28 15:25:43,244 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:43,244 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 94 to run in 60 seconds 2017-08-28 15:25:43,279 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 13 2017-08-28 15:25:43,283 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:25:43,296 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-08-28 15:25:44,056 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:44,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:44,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:44,067 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-08-28 15:25:45,230 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 93 2017-08-28 15:25:45,233 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:45,240 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:45,241 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:45,241 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:45,241 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node3 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:45,241 | DEBUG | node.py (1976) | checkPerformance | Node3's master has higher performance than backups 2017-08-28 15:25:45,243 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-08-28 15:25:47,487 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:25:47,510 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on request 1503933947440487 2017-08-28 15:25:47,514 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,515 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,521 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,536 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,537 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating 6XiqM9DwYDgFnmyQQBFr72 request 1503933947440487 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,545 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:47,546 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since not finalised 2017-08-28 15:25:47,555 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,567 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node2 2017-08-28 15:25:47,580 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node2 2017-08-28 15:25:47,581 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node4 2017-08-28 15:25:47,587 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node4 2017-08-28 15:25:47,590 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node1 2017-08-28 15:25:47,596 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node1 2017-08-28 15:25:47,610 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:25:47,629 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,630 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,631 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}}, 'Node2') 2017-08-28 15:25:47,637 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,639 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,654 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,654 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}}, 'Node4') 2017-08-28 15:25:47,655 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,665 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,673 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,674 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}}, 'Node1') 2017-08-28 15:25:47,683 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,685 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,692 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,699 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) to 0 replicas 2017-08-28 15:25:47,700 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,714 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,715 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since already forwarded 2017-08-28 15:25:47,724 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,730 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,732 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since already forwarded 2017-08-28 15:25:47,743 | DEBUG | replica.py ( 665) | create3PCBatch | Node3:1 creating batch 3 for ledger 0 with state root None 2017-08-28 15:25:47,764 | DEBUG | replica.py ( 698) | create3PCBatch | Node3:1 created a PRE-PREPARE with 1 requests for ledger 0 2017-08-28 15:25:47,764 | DEBUG | replica.py (2133) | send | Node3:1 sending PrePrepare 2017-08-28 15:25:47,765 | TRACE | replica.py (2134) | send | Node3:1 sending PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} 2017-08-28 15:25:47,765 | DEBUG | replica.py ( 612) | trackBatches | Node3:1 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} with state root None 2017-08-28 15:25:47,799 | DEBUG | node.py (2566) | send | Node3 sending message PREPREPARE{'ledgerId': 0, 'discarded': 1, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:47,812 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node2 2017-08-28 15:25:47,813 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node2 2017-08-28 15:25:47,813 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node4 2017-08-28 15:25:47,814 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node4 2017-08-28 15:25:47,829 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node1 2017-08-28 15:25:47,830 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":null,"discarded":1,"op":"PREPREPARE","viewNo":1,"ppSeqNo":3,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"ledgerId":0,"instId":1,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":null}' to Node1 2017-08-28 15:25:47,842 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:47,843 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]]} 2017-08-28 15:25:47,855 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'ppSeqNo': 3, 'op': 'PREPREPARE', 'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]]}, 'Node2') 2017-08-28 15:25:47,856 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]]} 2017-08-28 15:25:47,858 | DEBUG | replica.py ( 821) | processPrePrepare | Node3:0 received PRE-PREPARE(1, 3) from Node2:0 at 250244.007495597 2017-08-28 15:25:47,879 | DEBUG | replica.py (1034) | validate_pre_prepare | Node3:0 state root before processing PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:25:47,886 | WARNING | replica.py ( 655) | processReqDuringBatch | Node3:0 encountered exception UnauthorizedClientRequest('6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node',) while processing Request: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}, will reject 2017-08-28 15:25:47,893 | DEBUG | replica.py ( 934) | doPrepare | Node3:0 Sending PREPARE(1, 3) at 250244.042256374 2017-08-28 15:25:47,894 | DEBUG | replica.py (2133) | send | Node3:0 sending Prepare 2017-08-28 15:25:47,896 | TRACE | replica.py (2134) | send | Node3:0 sending PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,907 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,908 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:47,908 | DEBUG | replica.py ( 840) | processPrePrepare | Node3:0 processed incoming PRE-PREPARE(1, 3) 2017-08-28 15:25:47,930 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REJECT","reqId":1503933947440487,"reason":"client request invalid: UnauthorizedClientRequest(\'6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node\',)","identifier":"6XiqM9DwYDgFnmyQQBFr72"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,947 | DEBUG | node.py (2566) | send | Node3 sending message PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:47,965 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:25:47,966 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 2017-08-28 15:25:47,966 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:25:47,973 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node4 2017-08-28 15:25:47,974 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:25:47,974 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1,"op":"PREPARE","ppSeqNo":3,"instId":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node1 2017-08-28 15:25:47,985 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:47,996 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,003 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'ppSeqNo': 3, 'op': 'PREPARE', 'ppTime': 1503933947, 'stateRootHash': None}, 'Node2') 2017-08-28 15:25:48,004 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,018 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 3) from Node2:1 2017-08-28 15:25:48,029 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,030 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,039 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:48,044 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}'], 'signature': None} 2017-08-28 15:25:48,045 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}'], 'signature': None}, 'Node1') 2017-08-28 15:25:48,059 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}'], 'signature': None} 2017-08-28 15:25:48,060 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,062 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPARE', 'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,070 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,071 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,071 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPARE', 'ppTime': 1503933947, 'stateRootHash': None, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,079 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,095 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 3) from Node1:0 2017-08-28 15:25:48,118 | DEBUG | replica.py ( 953) | doCommit | Node3:0 Sending COMMIT(1, 3) at 250244.267200347 2017-08-28 15:25:48,118 | DEBUG | replica.py (2133) | send | Node3:0 sending Commit 2017-08-28 15:25:48,119 | TRACE | replica.py (2134) | send | Node3:0 sending COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,119 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:25:48,123 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,127 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 3) from Node1:1 2017-08-28 15:25:48,128 | DEBUG | replica.py ( 953) | doCommit | Node3:1 Sending COMMIT(1, 3) at 250244.27699143 2017-08-28 15:25:48,128 | DEBUG | replica.py (2133) | send | Node3:1 sending Commit 2017-08-28 15:25:48,134 | TRACE | replica.py (2134) | send | Node3:1 sending COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,135 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:25:48,135 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,149 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:48,149 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}'], 'signature': None} 2017-08-28 15:25:48,162 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:48,162 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}'], 'signature': None} 2017-08-28 15:25:48,172 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,172 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPARE', 'ppTime': 1503933947, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,172 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,181 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,181 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPARE', 'ppTime': 1503933947, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,181 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,202 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:48,208 | DEBUG | node.py (2566) | send | Node3 sending message COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:48,208 | DEBUG | replica.py ( 865) | processPrepare | Node3:0 received PREPARE(1, 3) from Node4:0 2017-08-28 15:25:48,208 | DEBUG | replica.py ( 917) | tryCommit | Node3:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,216 | DEBUG | replica.py ( 876) | processPrepare | Node3:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,217 | DEBUG | replica.py ( 865) | processPrepare | Node3:1 received PREPARE(1, 3) from Node4:1 2017-08-28 15:25:48,217 | DEBUG | replica.py ( 917) | tryCommit | Node3:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,227 | DEBUG | replica.py ( 876) | processPrepare | Node3:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,230 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:48,236 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,236 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node1') 2017-08-28 15:25:48,236 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,249 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:48,262 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:25:48,262 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:25:48,263 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node2 2017-08-28 15:25:48,267 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:48,274 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:25:48,274 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:25:48,275 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node4 2017-08-28 15:25:48,283 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:48,284 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', b'{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']) 2017-08-28 15:25:48,287 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' 2017-08-28 15:25:48,293 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":0}","{\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3,\\"instId\\":1}"],"signature":null}' to Node1 2017-08-28 15:25:48,295 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 3) from Node1:1 2017-08-28 15:25:48,314 | DEBUG | replica.py ( 929) | tryOrder | Node3:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} commits where f is 1 2017-08-28 15:25:48,314 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,325 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:25:48,326 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}'], 'signature': None} 2017-08-28 15:25:48,332 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}'], 'signature': None}, 'Node2') 2017-08-28 15:25:48,332 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}'], 'signature': None} 2017-08-28 15:25:48,332 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,333 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node2') 2017-08-28 15:25:48,339 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,344 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,344 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 1, 'viewNo': 1}, 'Node2') 2017-08-28 15:25:48,344 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,355 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:25:48,355 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:48,362 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}'], 'signature': None} 2017-08-28 15:25:48,363 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,363 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 1}, 'Node4') 2017-08-28 15:25:48,368 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,368 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,368 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 1}, 'Node4') 2017-08-28 15:25:48,368 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} 2017-08-28 15:25:48,369 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,381 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node1') 2017-08-28 15:25:48,382 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-08-28 15:25:48,408 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 3) from Node2:0 2017-08-28 15:25:48,408 | DEBUG | replica.py ( 929) | tryOrder | Node3:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} commits where f is 1 2017-08-28 15:25:48,408 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,409 | DEBUG | replica.py ( 894) | processCommit | Node3:0 received COMMIT(1, 3) from Node4:0 2017-08-28 15:25:48,415 | TRACE | replica.py ( 925) | tryOrder | Node3:0 returning request to node 2017-08-28 15:25:48,415 | DEBUG | replica.py (1437) | doOrder | Node3:0 ordering COMMIT (1, 3) 2017-08-28 15:25:48,421 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:0 set last ordered as (1, 3) 2017-08-28 15:25:48,421 | DEBUG | replica.py (2133) | send | Node3:0 sending Ordered 2017-08-28 15:25:48,422 | TRACE | replica.py (2134) | send | Node3:0 sending ORDERED{'ledgerId': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 1, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': []} 2017-08-28 15:25:48,427 | DEBUG | replica.py (1469) | order_3pc_key | Node3:0 ordered request (1, 3) 2017-08-28 15:25:48,437 | DEBUG | replica.py ( 905) | processCommit | Node3:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,437 | TRACE | message_processor.py ( 29) | discard | Node3:0 discarding message COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:25:48,438 | DEBUG | replica.py ( 894) | processCommit | Node3:1 received COMMIT(1, 3) from Node2:1 2017-08-28 15:25:48,438 | TRACE | replica.py ( 925) | tryOrder | Node3:1 returning request to node 2017-08-28 15:25:48,438 | DEBUG | replica.py (1437) | doOrder | Node3:1 ordering COMMIT (1, 3) 2017-08-28 15:25:48,440 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (1, 3) 2017-08-28 15:25:48,451 | DEBUG | replica.py (2133) | send | Node3:1 sending Ordered 2017-08-28 15:25:48,464 | TRACE | replica.py (2134) | send | Node3:1 sending ORDERED{'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'viewNo': 1, 'stateRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)]} 2017-08-28 15:25:48,464 | DEBUG | replica.py (1469) | order_3pc_key | Node3:1 ordered request (1, 3) 2017-08-28 15:25:48,464 | DEBUG | replica.py ( 905) | processCommit | Node3:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,466 | TRACE | message_processor.py ( 29) | discard | Node3:1 discarding message COMMIT{'ppSeqNo': 3, 'instId': 1, 'viewNo': 1} because already ordered 3 phase message 2017-08-28 15:25:48,486 | DEBUG | node.py (1828) | processOrdered | Node3 executing Ordered batch 1 3 of 0 requests 2017-08-28 15:25:48,487 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:25:48,502 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:48,502 | TRACE | node.py (1839) | processOrdered | Node3 got ordered requests from backup replica 1 2017-08-28 15:25:48,547 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.062 seconds 2017-08-28 15:25:52,679 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:52,680 | DEBUG | zstack.py ( 661) | handlePingPong | Node3 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,691 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,693 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node3 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:52,703 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:53,662 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:53,675 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,681 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 25, 'viewNo': 2}, 'Node4') 2017-08-28 15:25:53,682 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,682 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node4 2017-08-28 15:25:53,689 | DEBUG | node.py (1939) | do_view_change_if_possible | Node3 has no quorum for view 2 2017-08-28 15:25:53,690 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node3 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:25:53,690 | INFO | node.py (1920) | processInstanceChange | VIEW CHANGE: Node3 found master degraded after receiving instance change message from Node4 2017-08-28 15:25:53,694 | DEBUG | throttler.py ( 31) | acquire | now: 250249.84310764, len(actionsLog): 0 2017-08-28 15:25:53,694 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:25:53,694 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:25:53,710 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 2 since Primary of master protocol instance degraded the performance 2017-08-28 15:25:53,710 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249459.128602697, 249459.134965826] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 2.1939092739776243} master request latencies: {} client avg request latencies: [{}, {'V4SGRU86Z58d6TV7PBUe6f': (2, 0.6954048164916458), '6XiqM9DwYDgFnmyQQBFr72': (1, 0.8030996409943327)}] throughput: {0: 0, 1: 1.3674220878609573} master throughput: 0 total requests: 6 avg backup throughput: 1.3674220878609573 master throughput ratio: 0.0 2017-08-28 15:25:53,719 | DEBUG | node.py (2566) | send | Node3 sending message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:53,724 | DEBUG | node.py (1939) | do_view_change_if_possible | Node3 has no quorum for view 2 2017-08-28 15:25:53,734 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node2 2017-08-28 15:25:53,734 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node2 2017-08-28 15:25:53,740 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node4 2017-08-28 15:25:53,741 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node4 2017-08-28 15:25:53,741 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node1 2017-08-28 15:25:53,742 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"INSTANCE_CHANGE","reason":25,"viewNo":2}' to Node1 2017-08-28 15:25:53,760 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:53,761 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,772 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 25, 'viewNo': 2}, 'Node2') 2017-08-28 15:25:53,772 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,776 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,776 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'INSTANCE_CHANGE', 'reason': 25, 'viewNo': 2}, 'Node1') 2017-08-28 15:25:53,777 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,791 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node2 2017-08-28 15:25:53,792 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 2 from 1 2017-08-28 15:25:53,792 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action _check_view_change_completed with id 96 to run in 60 seconds 2017-08-28 15:25:53,792 | DEBUG | replica.py ( 453) | on_view_change_start | Node3:0 setting last prepared for master to (1, 3) 2017-08-28 15:25:53,792 | DEBUG | node.py (2127) | startViewChange | Node3 resetting monitor stats after view change 2017-08-28 15:25:53,792 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:25:53,793 | DEBUG | replica.py ( 406) | primaryName | Node3:0 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,793 | DEBUG | replica.py ( 406) | primaryName | Node3:1 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,793 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node3 changed to view 2, will start catchup now 2017-08-28 15:25:53,793 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,794 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:53,794 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,794 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:53,794 | DEBUG | node.py (2173) | start_catchup | Node3 reverted 0 batches before starting catch up 2017-08-28 15:25:53,794 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:25:53,794 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 0 2017-08-28 15:25:53,794 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:25:53,795 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 0 2017-08-28 15:25:53,817 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:53,821 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 0 2017-08-28 15:25:53,821 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:25:53,821 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 0 2017-08-28 15:25:53,822 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-08-28 15:25:53,827 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:53,828 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:25:53,828 | DEBUG | node.py (1927) | processInstanceChange | Node3 received instance change message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:25:53,840 | DEBUG | node.py (1896) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node1 2017-08-28 15:25:53,841 | INFO | message_processor.py ( 29) | discard | Node3 discarding message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} because Received instance change request with view no 2 which is not more than its view no 2 2017-08-28 15:25:53,842 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:53,855 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:53,856 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:53,856 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:25:53,867 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:53,871 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:53,872 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:53,872 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:25:53,873 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:53,874 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:53,875 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:53,883 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:25:53,890 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:53,905 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:53,906 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:53,906 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:53,915 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:53,915 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:53,921 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:53,922 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:53,922 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:53,922 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:53,931 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:25:53,946 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:25:53,960 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:53,961 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:25:53,961 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:53,968 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:25:53,973 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:53,992 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,003 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:54,003 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,004 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,004 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:54,005 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,005 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,014 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:54,014 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,014 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:54,024 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:54,025 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:54,025 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,026 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:54,027 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,041 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,041 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:54,041 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,042 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,051 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,053 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,065 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,079 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,083 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:25:54,084 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:54,093 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:25:54,093 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,093 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']) 2017-08-28 15:25:54,104 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:54,108 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null}}"],"signature":null}' to Node4 2017-08-28 15:25:54,111 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:54,129 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,130 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,144 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,151 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,152 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node1') 2017-08-28 15:25:54,152 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,152 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,158 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node1') 2017-08-28 15:25:54,159 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,165 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,166 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:54,166 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None} 2017-08-28 15:25:54,184 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,184 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node2') 2017-08-28 15:25:54,184 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,185 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,187 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node2') 2017-08-28 15:25:54,188 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,188 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node1 2017-08-28 15:25:54,203 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,204 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,204 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node1 2017-08-28 15:25:54,210 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,211 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,216 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node2 2017-08-28 15:25:54,216 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,217 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,227 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 0 is latest 2017-08-28 15:25:54,227 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} that it does not need catchup 2017-08-28 15:25:54,228 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,228 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,234 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,238 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,238 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,239 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:25:54,244 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:25:54,244 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:25:54,244 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,245 | DEBUG | node.py ( 923) | checkInstances | Node3 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node4', 'Node1'} 2017-08-28 15:25:54,251 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,252 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 2 2017-08-28 15:25:54,252 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,258 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 2 2017-08-28 15:25:54,259 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:54,263 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 2 2017-08-28 15:25:54,263 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,264 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 2 2017-08-28 15:25:54,264 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:25:54,266 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:25:54,275 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node2 2017-08-28 15:25:54,276 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,283 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,295 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,296 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:54,296 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:54,302 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:25:54,307 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,308 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:54,308 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:54,319 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:25:54,322 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,322 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:54,322 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:54,334 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:25:54,341 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:54,346 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:54,359 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:54,360 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:54,363 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,376 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node4') 2017-08-28 15:25:54,376 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,379 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,380 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}}, 'Node4') 2017-08-28 15:25:54,380 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 0}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None}} 2017-08-28 15:25:54,391 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,392 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,392 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,402 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,403 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:54,403 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,410 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,410 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:54,410 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,417 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node4 2017-08-28 15:25:54,422 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,422 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,422 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:25:54,424 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': None} from Node4 2017-08-28 15:25:54,437 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,437 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,438 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:25:54,444 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,452 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,472 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,473 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:25:54,476 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:54,487 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-08-28 15:25:54,501 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:54,502 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,514 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:54,514 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,515 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,520 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:54,520 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,520 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,532 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:54,532 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,533 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:54,534 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:54,544 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:54,554 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,556 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:54,556 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,557 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,564 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:54,564 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:54,565 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,579 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,590 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,592 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,605 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,625 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:25:54,626 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:54,627 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node2 2017-08-28 15:25:54,638 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,638 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-08-28 15:25:54,639 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-08-28 15:25:54,655 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node4 2017-08-28 15:25:54,674 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:25:54,683 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,689 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:54,689 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,700 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,701 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:25:54,709 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,710 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,710 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node2') 2017-08-28 15:25:54,723 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,724 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,724 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,725 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-08-28 15:25:54,741 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,741 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:25:54,741 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,742 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,749 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node1') 2017-08-28 15:25:54,752 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,753 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:54,762 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:54,769 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:54,769 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,773 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:25:54,773 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,773 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,781 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}}, 'Node4') 2017-08-28 15:25:54,781 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None}} 2017-08-28 15:25:54,782 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:25:54,786 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,795 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,796 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node2 2017-08-28 15:25:54,807 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,814 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,822 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:25:54,823 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,830 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,833 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:25:54,833 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} that it does not need catchup 2017-08-28 15:25:54,834 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,846 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,849 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,850 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,850 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,858 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:25:54,859 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:25:54,859 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:25:54,860 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,860 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:25:54,870 | DEBUG | upgrader.py ( 208) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya processing config ledger for any upgrades 2017-08-28 15:25:54,870 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,879 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 1 2017-08-28 15:25:54,880 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,889 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 1 2017-08-28 15:25:54,889 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:54,889 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 1 2017-08-28 15:25:54,900 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,901 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 1 2017-08-28 15:25:54,907 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:25:54,908 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-08-28 15:25:54,908 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:25:54,909 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node1 2017-08-28 15:25:54,917 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,918 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,929 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:25:54,930 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,940 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,940 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:25:54,947 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None} from Node4 2017-08-28 15:25:54,950 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,950 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,951 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:25:54,962 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,977 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:54,977 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:54,983 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node2 2017-08-28 15:25:54,984 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,984 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:54,996 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:54,996 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node4 2017-08-28 15:25:54,997 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:55,008 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']) 2017-08-28 15:25:55,008 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' 2017-08-28 15:25:55,009 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}"],"signature":null}' to Node1 2017-08-28 15:25:55,042 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 3 messages through listener 2017-08-28 15:25:55,055 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,056 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:55,059 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,060 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,060 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:55,060 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,072 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,072 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1') 2017-08-28 15:25:55,073 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,073 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,073 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:55,084 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,084 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,085 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:55,095 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,096 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,096 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2') 2017-08-28 15:25:55,096 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,104 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:55,112 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:55,112 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-08-28 15:25:55,113 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,124 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:55,124 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,130 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,130 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node4') 2017-08-28 15:25:55,130 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} 2017-08-28 15:25:55,146 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:25:55,157 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:25:55,164 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:25:55,165 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:25:55,176 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,187 | DEBUG | node.py (2566) | send | Node3 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,208 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:55,212 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:25:55,212 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:25:55,222 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node2 2017-08-28 15:25:55,222 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:55,233 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:25:55,234 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:25:55,241 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node4 2017-08-28 15:25:55,242 | DEBUG | batched.py ( 99) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:55,252 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']) 2017-08-28 15:25:55,252 | TRACE | batched.py ( 108) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' 2017-08-28 15:25:55,262 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0}}"],"signature":null}' to Node1 2017-08-28 15:25:55,281 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:55,290 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,305 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node2') 2017-08-28 15:25:55,309 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,312 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,313 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node2') 2017-08-28 15:25:55,316 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,330 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,330 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node2') 2017-08-28 15:25:55,331 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,341 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,342 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:55,353 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-08-28 15:25:55,354 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,360 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node1') 2017-08-28 15:25:55,361 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,361 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,373 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node1') 2017-08-28 15:25:55,375 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,376 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node2 2017-08-28 15:25:55,383 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,384 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,393 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node2 2017-08-28 15:25:55,396 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,396 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,407 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node1 2017-08-28 15:25:55,408 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,408 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,415 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:25:55,416 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} that it does not need catchup 2017-08-28 15:25:55,416 | DEBUG | node.py (1461) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-08-28 15:25:55,423 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,423 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:55,424 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,424 | DEBUG | node.py (1869) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:55,433 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:25:55,434 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:25:55,434 | INFO | node.py (1471) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:25:55,442 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,443 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:25:55,447 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,448 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-08-28 15:25:55,448 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-08-28 15:25:55,457 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 3 ViewChangeDone messages 2017-08-28 15:25:55,459 | DEBUG | node.py (1550) | caught_up_for_current_view | Node3 does not have view change quorum for view 2 2017-08-28 15:25:55,461 | DEBUG | node.py (1534) | is_catchup_needed | Node3 is not caught up for the current view 2 2017-08-28 15:25:55,475 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,476 | DEBUG | node.py (1539) | is_catchup_needed | Node3 ordered till last prepared certificate 2017-08-28 15:25:55,476 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-08-28 15:25:55,495 | DEBUG | primary_decider.py ( 131) | send | Node3's elector sending VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 2} 2017-08-28 15:25:55,496 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 2 ViewChangeDone messages 2017-08-28 15:25:55,510 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,510 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node1 2017-08-28 15:25:55,511 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,512 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,523 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 95 2017-08-28 15:25:55,523 | TRACE | node.py (1959) | checkPerformance | Node3 checking its performance 2017-08-28 15:25:55,527 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-08-28 15:25:55,528 | DEBUG | node.py (2566) | send | Node3 sending message VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 2} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:55,544 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node2 2017-08-28 15:25:55,545 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node2 2017-08-28 15:25:55,551 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node4 2017-08-28 15:25:55,552 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node4 2017-08-28 15:25:55,552 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node1 2017-08-28 15:25:55,563 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"VIEW_CHANGE_DONE","name":"Node3","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":2}' to Node1 2017-08-28 15:25:55,579 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:55,584 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:55,585 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node4') 2017-08-28 15:25:55,595 | DEBUG | node.py (1254) | unpackNodeMsg | Node3 processing a batch BATCH{'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-08-28 15:25:55,604 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,604 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node4') 2017-08-28 15:25:55,607 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,608 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,608 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}}, 'Node4') 2017-08-28 15:25:55,608 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0}} 2017-08-28 15:25:55,621 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,622 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node2') 2017-08-28 15:25:55,622 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,623 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node4 2017-08-28 15:25:55,629 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,629 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,629 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:25:55,636 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ledgerId': 1, 'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'viewNo': 0} from Node4 2017-08-28 15:25:55,644 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,644 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,644 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:25:55,653 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node2') 2017-08-28 15:25:55,655 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,669 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-08-28 15:25:55,669 | DEBUG | primary_selector.py ( 265) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,684 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:55,685 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,687 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node1') 2017-08-28 15:25:55,695 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,696 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,697 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node4') 2017-08-28 15:25:55,697 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,710 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node1') 2017-08-28 15:25:55,714 | DEBUG | node.py (1187) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2}, 'Node4') 2017-08-28 15:25:55,720 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,731 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node3 got view change quorum (3 >= 3) 2017-08-28 15:25:55,732 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node3 received ViewChangeDone from primary Node3 2017-08-28 15:25:55,732 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node3 found acceptable primary Node3 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:25:55,733 | DEBUG | primary_selector.py ( 280) | _startSelection | Node3 starting selection 2017-08-28 15:25:55,741 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node3:0 for instance 0 (view 2) 2017-08-28 15:25:55,741 | INFO | node.py ( 503) | start_participating | Node3 started participating 2017-08-28 15:25:55,741 | DEBUG | replica.py ( 406) | primaryName | Node3:0 setting primaryName for view no 2 to: Node3:0 2017-08-28 15:25:55,742 | DEBUG | replica.py (1678) | _gc | Node3:0 cleaning up till (1, 3) 2017-08-28 15:25:55,742 | DEBUG | replica.py (1693) | _gc | Node3:0 found 3 3-phase keys to clean 2017-08-28 15:25:55,754 | DEBUG | replica.py (1695) | _gc | Node3:0 found 3 request keys to clean 2017-08-28 15:25:55,754 | DEBUG | replica.py ( 311) | h | Node3:0 set watermarks as 0 300 2017-08-28 15:25:55,763 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,764 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node4:1 for instance 1 (view 2) 2017-08-28 15:25:55,764 | DEBUG | replica.py ( 406) | primaryName | Node3:1 setting primaryName for view no 2 to: Node4:1 2017-08-28 15:25:55,765 | DEBUG | replica.py (1678) | _gc | Node3:1 cleaning up till (1, 3) 2017-08-28 15:25:55,774 | DEBUG | replica.py (1693) | _gc | Node3:1 found 3 3-phase keys to clean 2017-08-28 15:25:55,774 | DEBUG | replica.py (1695) | _gc | Node3:1 found 3 request keys to clean 2017-08-28 15:25:55,774 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) from previous checkpoints 2017-08-28 15:25:55,790 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) from previous checkpoints 2017-08-28 15:25:55,795 | DEBUG | replica.py (1716) | _gc | Node3:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) from previous checkpoints 2017-08-28 15:25:55,796 | DEBUG | replica.py ( 311) | h | Node3:1 set watermarks as 0 300 2017-08-28 15:25:55,796 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node3:1 Setting last ordered for non-master as (1, 3) 2017-08-28 15:25:55,796 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node3:1 set last ordered as (2, 0) 2017-08-28 15:25:55,797 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node3:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,807 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} 2017-08-28 15:25:55,808 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'name': 'Node3', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 2} because it already decided primary which is Node3:0 2017-08-28 15:25:55,834 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 2.173 seconds 2017-08-28 15:25:57,411 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3C got 1 messages through listener 2017-08-28 15:25:57,422 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on request 1503933957374783 2017-08-28 15:25:57,436 | TRACE | node.py (1370) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,437 | DEBUG | node.py (1416) | processClientInBox | Node3C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,444 | DEBUG | node.py (1690) | processRequest | Node3 received client request: SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,446 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,459 | DEBUG | propagator.py ( 148) | propagate | Node3 propagating Rhx2qwaeiGyhU9vn4ynHSS request 1503933957374783 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,460 | DEBUG | node.py (2566) | send | Node3 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:57,467 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since not finalised 2017-08-28 15:25:57,471 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REQACK","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:57,484 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node2 2017-08-28 15:25:57,485 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node2 2017-08-28 15:25:57,496 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node4 2017-08-28 15:25:57,506 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node4 2017-08-28 15:25:57,507 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node1 2017-08-28 15:25:57,516 | TRACE | batched.py ( 95) | flushOutBoxes | Node3 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"operation":{"type":"0","data":{"node_port":9701,"node_ip":"10.0.0.105","client_port":9702,"client_ip":"10.0.0.105","services":["VALIDATOR"],"alias":"Node5"},"dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"},"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node1 2017-08-28 15:25:57,518 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 1 messages through listener 2017-08-28 15:25:57,543 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,544 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,544 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}}, 'Node1') 2017-08-28 15:25:57,558 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,559 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,566 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,567 | DEBUG | propagator.py ( 213) | forward | Node3 forwarding request ('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783) to 0 replicas 2017-08-28 15:25:57,592 | DEBUG | replica.py ( 665) | create3PCBatch | Node3:0 creating batch 1 for ledger 0 with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:57,603 | WARNING | replica.py ( 655) | processReqDuringBatch | Node3:0 encountered exception UnauthorizedClientRequest('Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot add a new node',) while processing Request: {'operation': {'type': '0', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}, will reject 2017-08-28 15:25:57,605 | DEBUG | replica.py ( 698) | create3PCBatch | Node3:0 created a PRE-PREPARE with 0 requests for ledger 0 2017-08-28 15:25:57,606 | DEBUG | replica.py (2133) | send | Node3:0 sending PrePrepare 2017-08-28 15:25:57,614 | TRACE | replica.py (2134) | send | Node3:0 sending PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933957, 'ppSeqNo': 1, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)]} 2017-08-28 15:25:57,615 | DEBUG | replica.py ( 612) | trackBatches | Node3:0 tracking batch for PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933957, 'ppSeqNo': 1, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)]} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:57,622 | TRACE | zstack.py ( 477) | _receiveFromListener | Node3 got 2 messages through listener 2017-08-28 15:25:57,642 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,643 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,649 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}}, 'Node4') 2017-08-28 15:25:57,657 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,668 | DEBUG | node.py (2217) | verifySignature | Node3 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,669 | DEBUG | node.py (1239) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,675 | DEBUG | node.py (1201) | handleOneNodeMsg | Node3 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}}, 'Node2') 2017-08-28 15:25:57,676 | DEBUG | node.py (1268) | postToNodeInBox | Node3 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,677 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,690 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,690 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since already forwarded 2017-08-28 15:25:57,708 | DEBUG | node.py (1738) | processPropagate | Node Node3 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,717 | TRACE | propagator.py ( 141) | propagate | Node3 already propagated SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,717 | DEBUG | propagator.py ( 245) | tryForwarding | Node3 not forwarding request SafeRequest: {'operation': {'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_port': 9701, 'node_ip': '10.0.0.105', 'client_port': 9702, 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'alias': 'Node5'}, 'type': '0'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since already forwarded 2017-08-28 15:25:57,737 | TRACE | zstack.py ( 752) | transmitThroughListener | Node3C transmitting b'{"op":"REJECT","reqId":1503933957374783,"reason":"client request invalid: UnauthorizedClientRequest(\'Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot add a new node\',)","identifier":"Rhx2qwaeiGyhU9vn4ynHSS"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:57,744 | DEBUG | node.py (2566) | send | Node3 sending message PREPREPARE{'ledgerId': 0, 'discarded': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933957, 'ppSeqNo': 1, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)]} to all recipients: ['Node2', 'Node4', 'Node1'] 2017-08-28 15:25:57,763 | DEBUG | zstack.py ( 721) | transmit | Node3 transmitting message b'{"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","discarded":0,"op":"PREPREPARE","viewNo":2,"ppSeqNo":1,"reqIdr":[["Rhx2qwaeiGyhU9vn4ynHSS",1503933957374783]],"ledgerId":0,"instId":0,"digest":"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166","ppTime":1503933957,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m"}' to Node2 20