2017-08-28 15:12:10,213 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node2.log 2017-08-28 15:12:10,214 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-08-28 15:12:12,621 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-08-28 15:12:12,688 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-08-28 15:12:12,805 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-08-28 15:12:12,817 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-08-28 15:12:14,632 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-08-28 15:12:14,633 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-08-28 15:12:14,665 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:14,693 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:14,749 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.055852524004876614 seconds 2017-08-28 15:12:14,805 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node2 2017-08-28 15:12:14,832 | INFO | node.py (2378) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-08-28 15:12:14,923 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:14,950 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:15,006 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.05554549800581299 seconds 2017-08-28 15:12:15,008 | INFO | node.py (2378) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-08-28 15:12:15,065 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:15,065 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-08-28 15:12:15,066 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:15,066 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:15,066 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:12:15,066 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-08-28 15:12:15,067 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-08-28 15:12:15,067 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-08-28 15:12:15,068 | DEBUG | replica.py ( 311) | h | Node2:0 set watermarks as 0 300 2017-08-28 15:12:15,072 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:0 to instance 0 (master) 2017-08-28 15:12:15,072 | DEBUG | replica.py ( 311) | h | Node2:1 set watermarks as 0 300 2017-08-28 15:12:15,073 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:1 to instance 1 (backup) 2017-08-28 15:12:15,073 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-08-28 15:12:15,073 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-08-28 15:12:15,074 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-08-28 15:12:15,074 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-08-28 15:12:15,074 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-08-28 15:12:15,075 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-08-28 15:12:15,075 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-08-28 15:12:15,075 | DEBUG | node.py ( 337) | __init__ | total plugins loaded in node: 0 2017-08-28 15:12:15,151 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:12:15,180 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:12:15,234 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0545334109920077 seconds 2017-08-28 15:12:15,263 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node2 has no upgrade events 2017-08-28 15:12:15,263 | INFO | node.py (2378) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-08-28 15:12:15,264 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from stopped to starting 2017-08-28 15:12:15,264 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:15,264 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:15,264 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:12:15,265 | DEBUG | zstack.py ( 319) | start | Node2 starting with restricted as True and reSetupAuth as True 2017-08-28 15:12:15,265 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-08-28 15:12:15,265 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:12:15,267 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node2/public_keys] 2017-08-28 15:12:15,269 | DEBUG | zstack.py ( 347) | open | Node2 will bind its listener at 9703 2017-08-28 15:12:15,269 | INFO | stacks.py ( 84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703 2017-08-28 15:12:15,270 | DEBUG | zstack.py ( 319) | start | Node2C starting with restricted as False and reSetupAuth as True 2017-08-28 15:12:15,270 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-08-28 15:12:15,271 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:12:15,271 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-08-28 15:12:15,272 | DEBUG | zstack.py ( 347) | open | Node2C will bind its listener at 9704 2017-08-28 15:12:15,272 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-08-28 15:12:15,272 | INFO | node.py ( 612) | start | Node2 first time running... 2017-08-28 15:12:15,276 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node2 found the following missing connections: Node4, Node1, Node3 2017-08-28 15:12:15,277 | TRACE | remote.py ( 84) | connect | connecting socket 78 34254896 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:15,277 | INFO | zstack.py ( 588) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707 2017-08-28 15:12:15,278 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node4 2017-08-28 15:12:15,282 | TRACE | remote.py ( 84) | connect | connecting socket 79 34359632 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-08-28 15:12:15,282 | INFO | zstack.py ( 588) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701 2017-08-28 15:12:15,283 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node1 2017-08-28 15:12:15,291 | TRACE | remote.py ( 84) | connect | connecting socket 81 34396816 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:15,292 | INFO | zstack.py ( 588) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705 2017-08-28 15:12:15,292 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node3 2017-08-28 15:12:15,296 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:12:15,311 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node4 2017-08-28 15:12:15,312 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node4 2017-08-28 15:12:15,312 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node1 2017-08-28 15:12:15,312 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-08-28 15:12:15,312 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node3 2017-08-28 15:12:15,313 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-08-28 15:12:15,394 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:12:15,395 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from Node1 2017-08-28 15:12:15,395 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged Node1 2017-08-28 15:12:15,396 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from Node1 2017-08-28 15:12:15,396 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged Node1 2017-08-28 15:12:15,397 | DEBUG | zstack.py ( 667) | handlePingPong | Node2 got pong from Node1 2017-08-28 15:12:15,398 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:15,398 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'po', b'po']) 2017-08-28 15:12:15,399 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["po","po"],"signature":null}' 2017-08-28 15:12:15,399 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["po","po"],"signature":null}' to Node1 2017-08-28 15:12:15,400 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from set() to {'Node1'} 2017-08-28 15:12:15,400 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-08-28 15:12:15,400 | DEBUG | node.py (2566) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} to 1 recipients: ['Node1'] 2017-08-28 15:12:15,411 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}' to Node1 2017-08-28 15:12:15,412 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}' to Node1 2017-08-28 15:12:15,425 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:15,426 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:15,426 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'op': 'LEDGER_STATUS', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'Node1') 2017-08-28 15:12:15,426 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:15,430 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node1 2017-08-28 15:12:15,430 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:15,430 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:25,083 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-08-28 15:12:25,084 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:12:25,084 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-08-28 15:12:30,287 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:30,292 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:12:30,292 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:30,295 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node4 2017-08-28 15:12:30,297 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node3 2017-08-28 15:12:30,297 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:12:30,314 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node4 2017-08-28 15:12:30,315 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node4 2017-08-28 15:12:30,315 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node3 2017-08-28 15:12:30,315 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-08-28 15:12:35,093 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 4 2017-08-28 15:12:35,094 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:12:35,094 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-08-28 15:12:43,533 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,631 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from Node3 2017-08-28 15:12:43,631 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged Node3 2017-08-28 15:12:43,632 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'po' to Node3 2017-08-28 15:12:43,633 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'po' to Node3 2017-08-28 15:12:43,642 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.111 seconds 2017-08-28 15:12:43,654 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,663 | DEBUG | zstack.py ( 667) | handlePingPong | Node2 got pong from Node3 2017-08-28 15:12:43,663 | DEBUG | zstack.py ( 667) | handlePingPong | Node2 got pong from Node3 2017-08-28 15:12:43,669 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1'} to {'Node1', 'Node3'} 2017-08-28 15:12:43,671 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-08-28 15:12:43,672 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from starting to started_hungry 2017-08-28 15:12:43,673 | DEBUG | node.py ( 923) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-08-28 15:12:43,673 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 0 2017-08-28 15:12:43,673 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'viewNo': 0, 'primary': []} to lagged node Node3 2017-08-28 15:12:43,679 | DEBUG | node.py (2566) | send | Node2 sending message CURRENT_STATE{'viewNo': 0, 'primary': []} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,680 | DEBUG | node.py (2566) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,695 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,696 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}']) 2017-08-28 15:12:43,702 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ledgerId\\":0,\\"ppSeqNo\\":null}"],"signature":null}' 2017-08-28 15:12:43,702 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ledgerId\\":0,\\"ppSeqNo\\":null}"],"signature":null}' to Node3 2017-08-28 15:12:43,715 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,716 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}']} 2017-08-28 15:12:43,717 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}']}, 'Node3') 2017-08-28 15:12:43,722 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}']} 2017-08-28 15:12:43,723 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: CURRENT_STATE{'viewNo': 0, 'primary': []} 2017-08-28 15:12:43,726 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'viewNo': 0, 'primary': []}, 'Node3') 2017-08-28 15:12:43,726 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': []} 2017-08-28 15:12:43,727 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:43,727 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'op': 'LEDGER_STATUS', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'Node3') 2017-08-28 15:12:43,727 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:43,728 | DEBUG | node.py ( 889) | process_current_state_message | Node2 processing current state CURRENT_STATE{'viewNo': 0, 'primary': []} from Node3 2017-08-28 15:12:43,728 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node3 2017-08-28 15:12:43,729 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,729 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,729 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:12:43,729 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:12:43,729 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,729 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,730 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,730 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,730 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,730 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:12:43,730 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:12:43,730 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:12:43,731 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,731 | DEBUG | node.py ( 923) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-08-28 15:12:43,731 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,747 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-08-28 15:12:43,748 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:12:43,748 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-08-28 15:12:43,748 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,748 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-08-28 15:12:43,749 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,749 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-08-28 15:12:43,749 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:12:43,749 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:12:43,751 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,751 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,769 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:12:43,769 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-08-28 15:12:43,769 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,769 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,770 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,779 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:12:43,779 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:12:43,780 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,780 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,780 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:12:43,780 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:12:43,799 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,799 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,799 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,800 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,800 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,800 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:12:43,801 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,801 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,801 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:12:43,801 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,802 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,802 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,816 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,816 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,816 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:12:43,817 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:12:43,818 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,831 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,832 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:12:43,832 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,832 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,832 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,832 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,832 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,832 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,833 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,834 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,834 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,856 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,857 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,857 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:12:43,861 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:12:43,863 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,872 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,872 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,872 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,873 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,873 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:12:43,873 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,873 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,873 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:12:43,873 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,874 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:12:43,884 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,884 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,884 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:12:43,885 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,885 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,902 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:43,907 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:12:43,908 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-08-28 15:12:43,908 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:12:43,908 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,908 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,909 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,909 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,909 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,909 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,910 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:12:43,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,916 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:12:43,916 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:12:43,916 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,916 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,917 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,917 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,917 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,917 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:12:43,917 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:12:43,917 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:12:43,917 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,918 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:12:43,918 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-08-28 15:12:43,918 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,918 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-08-28 15:12:43,918 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:12:43,918 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-08-28 15:12:43,918 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,919 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-08-28 15:12:43,938 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,938 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-08-28 15:12:43,938 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:12:43,939 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-08-28 15:12:43,939 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:12:43,940 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:12:43,941 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,941 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,943 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,943 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,943 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:43,943 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:12:43,943 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,955 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:43,955 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,956 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:43,960 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:12:43,960 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,960 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,961 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:43,965 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:12:43,968 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:12:43,969 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:43,969 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:43,969 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:43,969 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,969 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:12:43,970 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,970 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,970 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:12:43,970 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,970 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,971 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:12:43,971 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,971 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,971 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:43,971 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,983 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,983 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:43,983 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,984 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,985 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:12:43,985 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,986 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:44,003 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:12:44,003 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:44,007 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:44,008 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:12:44,011 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:44,012 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:44,012 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:44,012 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:12:44,028 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.314 seconds 2017-08-28 15:12:44,078 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:12:44,078 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:44,078 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:12:44,079 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:44,079 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,083 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:12:44,083 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,084 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,084 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:12:44,084 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,084 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']} 2017-08-28 15:12:44,084 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']}, 'Node3') 2017-08-28 15:12:44,084 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']} 2017-08-28 15:12:44,085 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,085 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,085 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,085 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,085 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,085 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,086 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} from Node1 2017-08-28 15:12:44,086 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,086 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,087 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} from Node1 2017-08-28 15:12:44,087 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,103 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,104 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} from Node3 2017-08-28 15:12:44,107 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,107 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,107 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:12:44,108 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:12:44,108 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:12:44,108 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,108 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:44,108 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,108 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:44,108 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:12:44,108 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:12:44,109 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:12:44,109 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,109 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:12:44,109 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,109 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:12:44,109 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:12:44,109 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-08-28 15:12:44,110 | DEBUG | node.py (1550) | caught_up_for_current_view | Node2 does not have view change quorum for view 0 2017-08-28 15:12:44,110 | DEBUG | node.py (1534) | is_catchup_needed | Node2 is not caught up for the current view 0 2017-08-28 15:12:44,110 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,110 | DEBUG | node.py (1539) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-08-28 15:12:44,110 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-08-28 15:12:44,111 | DEBUG | primary_decider.py ( 131) | send | Node2's elector sending VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} 2017-08-28 15:12:44,119 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-08-28 15:12:44,119 | DEBUG | primary_selector.py ( 265) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:12:44,120 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} from Node3 2017-08-28 15:12:44,120 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,120 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,121 | DEBUG | node.py (2566) | send | Node2 sending message VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:12:44,122 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:12:44,122 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:44,122 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:12:44,129 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:12:44,129 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:12:44,129 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:12:44,129 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:12:44,174 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:12:44,175 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,175 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0, 'name': 'Node1'}, 'Node1') 2017-08-28 15:12:44,175 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,176 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,176 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:12:44,176 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,177 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node1') 2017-08-28 15:12:44,177 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:12:44,262 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,262 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (2 >= 2) 2017-08-28 15:12:44,263 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node1 2017-08-28 15:12:44,263 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:12:44,263 | DEBUG | primary_selector.py ( 280) | _startSelection | Node2 starting selection 2017-08-28 15:12:44,264 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 0) 2017-08-28 15:12:44,264 | INFO | node.py ( 503) | start_participating | Node2 started participating 2017-08-28 15:12:44,264 | DEBUG | replica.py ( 406) | primaryName | Node2:0 setting primaryName for view no 0 to: Node1:0 2017-08-28 15:12:44,264 | DEBUG | replica.py (1678) | _gc | Node2:0 cleaning up till (0, 0) 2017-08-28 15:12:44,264 | DEBUG | replica.py (1693) | _gc | Node2:0 found 0 3-phase keys to clean 2017-08-28 15:12:44,264 | DEBUG | replica.py (1695) | _gc | Node2:0 found 0 request keys to clean 2017-08-28 15:12:44,264 | DEBUG | replica.py ( 311) | h | Node2:0 set watermarks as 0 300 2017-08-28 15:12:44,265 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,271 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 0) 2017-08-28 15:12:44,271 | DEBUG | replica.py ( 406) | primaryName | Node2:1 setting primaryName for view no 0 to: Node2:1 2017-08-28 15:12:44,272 | DEBUG | replica.py (1678) | _gc | Node2:1 cleaning up till (0, 0) 2017-08-28 15:12:44,272 | DEBUG | replica.py (1693) | _gc | Node2:1 found 0 3-phase keys to clean 2017-08-28 15:12:44,272 | DEBUG | replica.py (1695) | _gc | Node2:1 found 0 request keys to clean 2017-08-28 15:12:44,272 | DEBUG | replica.py ( 311) | h | Node2:1 set watermarks as 0 300 2017-08-28 15:12:44,272 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (0, 0) 2017-08-28 15:12:44,272 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 0) 2017-08-28 15:12:44,273 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,273 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:44,273 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} because it already decided primary which is Node1:0 2017-08-28 15:12:44,297 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.124 seconds 2017-08-28 15:12:45,094 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 5 2017-08-28 15:12:45,095 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:12:45,100 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:12:45,100 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:12:45,100 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:45,100 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:45,101 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:12:45,101 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-08-28 15:12:45,292 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:45,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:12:45,293 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:45,293 | DEBUG | zstack.py ( 603) | reconnectRemote | Node2 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:45,294 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:45,294 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 78 34254896 2017-08-28 15:12:45,295 | TRACE | remote.py ( 84) | connect | connecting socket 78 34254896 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:45,295 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 pinged Node4 2017-08-28 15:12:45,296 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:12:45,322 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'pi' to Node4 2017-08-28 15:12:45,322 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'pi' to Node4 2017-08-28 15:12:51,673 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:51,678 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from Node4 2017-08-28 15:12:51,678 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged Node4 2017-08-28 15:12:51,687 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'po' to Node4 2017-08-28 15:12:51,691 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'po' to Node4 2017-08-28 15:12:52,202 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:52,202 | DEBUG | zstack.py ( 667) | handlePingPong | Node2 got pong from Node4 2017-08-28 15:12:52,207 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3'} to {'Node4', 'Node1', 'Node3'} 2017-08-28 15:12:52,211 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node4 2017-08-28 15:12:52,212 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-08-28 15:12:52,212 | DEBUG | node.py ( 923) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node4', 'Node1', 'Node3'} 2017-08-28 15:12:52,213 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'viewNo': 0, 'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'))}]} to lagged node Node4 2017-08-28 15:12:52,219 | DEBUG | node.py (2566) | send | Node2 sending message CURRENT_STATE{'viewNo': 0, 'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'))}]} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,220 | DEBUG | node.py (2566) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,220 | DEBUG | node.py (2566) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,220 | DEBUG | node.py (2566) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,237 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,238 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}', b'{"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","op":"LEDGER_STATUS","txnSeqNo":9,"viewNo":null,"ledgerId":1,"ppSeqNo":null}', b'{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","txnSeqNo":0,"viewNo":null,"ledgerId":2,"ppSeqNo":null}']) 2017-08-28 15:12:52,241 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}]}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ledgerId\\":0,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ledgerId\\":1,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ledgerId\\":2,\\"ppSeqNo\\":null}"],"signature":null}' 2017-08-28 15:12:52,242 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}]}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ledgerId\\":0,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ledgerId\\":1,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ledgerId\\":2,\\"ppSeqNo\\":null}"],"signature":null}' to Node4 2017-08-28 15:12:52,459 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:52,460 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:12:52,463 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}']}, 'Node4') 2017-08-28 15:12:52,470 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:12:52,471 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: CURRENT_STATE{'viewNo': 0, 'primary': []} 2017-08-28 15:12:52,471 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'viewNo': 0, 'primary': []}, 'Node4') 2017-08-28 15:12:52,471 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': []} 2017-08-28 15:12:52,472 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:52,479 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'op': 'LEDGER_STATUS', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'Node4') 2017-08-28 15:12:52,479 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:52,480 | DEBUG | node.py ( 889) | process_current_state_message | Node2 processing current state CURRENT_STATE{'viewNo': 0, 'primary': []} from Node4 2017-08-28 15:12:52,495 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node4 2017-08-28 15:12:52,496 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,499 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,500 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:12:52,505 | DEBUG | zstack.py ( 661) | handlePingPong | Node2C got ping from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,505 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'po' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,507 | DEBUG | zstack.py ( 643) | sendPingPong | Node2C ponged b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,576 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:12:52,577 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:52,577 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} 2017-08-28 15:12:52,587 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,595 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,608 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null,"ledgerId":0,"ppSeqNo":null}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,619 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,620 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,816 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:52,820 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-08-28 15:12:52,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']}, 'Node4') 2017-08-28 15:12:52,827 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-08-28 15:12:52,835 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,836 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,836 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,842 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,843 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,847 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,847 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,847 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,850 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,851 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,855 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,855 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,856 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:52,860 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0, 'name': 'Node1'}, 'Node4') 2017-08-28 15:12:52,860 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:52,861 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,866 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,871 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,872 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'txnSeqNo': 9, 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,873 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node4') 2017-08-28 15:12:52,881 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:12:52,889 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node1', 'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} because it already decided primary which is Node1:0 2017-08-28 15:12:52,890 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,890 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:52,898 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:12:52,899 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:12:52,914 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.105 seconds 2017-08-28 15:12:53,377 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:12:53,383 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:12:53,384 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 1, 'reason': 26}, 'Node1') 2017-08-28 15:12:53,384 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:12:53,389 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} from Node1 2017-08-28 15:12:53,389 | DEBUG | node.py (1939) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-08-28 15:12:53,390 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:12:53,397 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:53,397 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:53,398 | DEBUG | node.py (1927) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:12:55,120 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-08-28 15:12:55,127 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:12:55,128 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:12:55,128 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:12:55,128 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:55,128 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:55,131 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:12:55,133 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-08-28 15:13:00,296 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:00,302 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:00,303 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:00,312 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:13:05,144 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-08-28 15:13:05,152 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:05,153 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:05,153 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:13:05,153 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:05,154 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:05,156 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:05,156 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-08-28 15:13:15,078 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 2 2017-08-28 15:13:15,078 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:13:15,078 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:15,079 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 9 to run in 60 seconds 2017-08-28 15:13:15,079 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-08-28 15:13:15,088 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:13:15,089 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-08-28 15:13:15,176 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-08-28 15:13:15,182 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:15,182 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:15,182 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:13:15,183 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:15,183 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:15,183 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:15,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-08-28 15:13:15,280 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action propose_view_change with id 3 2017-08-28 15:13:15,280 | TRACE | node.py (2078) | propose_view_change | The primary is already connected so view change will not be proposed 2017-08-28 15:13:15,297 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:15,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:15,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:15,300 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:13:20,686 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:13:20,698 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933200605368 2017-08-28 15:13:20,710 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,713 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,714 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,714 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,715 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933200605368 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,721 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:13:20,722 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} to its replicas since not finalised 2017-08-28 15:13:20,722 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:20,728 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node4 2017-08-28 15:13:20,728 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node4 2017-08-28 15:13:20,735 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node1 2017-08-28 15:13:20,739 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node1 2017-08-28 15:13:20,739 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node3 2017-08-28 15:13:20,740 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933200605368,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"}}}' to Node3 2017-08-28 15:13:20,742 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:13:20,753 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,761 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,761 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:13:20,763 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,764 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,773 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,773 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:13:20,773 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,774 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,779 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,779 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:13:20,780 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,780 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,785 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,785 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to 0 replicas 2017-08-28 15:13:20,786 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,791 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,792 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} to its replicas since already forwarded 2017-08-28 15:13:20,792 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933200605368, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:13:20,798 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} 2017-08-28 15:13:20,799 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933200605368, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}} to its replicas since already forwarded 2017-08-28 15:13:20,803 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 1 for ledger 1 with state root None 2017-08-28 15:13:20,813 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:13:20,813 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:13:20,814 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 1} 2017-08-28 15:13:20,814 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 1} with state root None 2017-08-28 15:13:20,824 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:13:20,826 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node4 2017-08-28 15:13:20,827 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node4 2017-08-28 15:13:20,827 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node1 2017-08-28 15:13:20,827 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node1 2017-08-28 15:13:20,827 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node3 2017-08-28 15:13:20,828 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933200,"discarded":1,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ledgerId":1}' to Node3 2017-08-28 15:13:20,857 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.197 seconds 2017-08-28 15:13:20,890 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:13:20,891 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1} 2017-08-28 15:13:20,891 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:20,894 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1} 2017-08-28 15:13:20,903 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 1) from Node1:0 at 249497.051969816 2017-08-28 15:13:20,908 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1} is b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[', 42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t 2017-08-28 15:13:20,910 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 1) at 249497.058731149 2017-08-28 15:13:20,921 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:13:20,921 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:20,921 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:20,922 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'discarded': 1, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1} with state root b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[' 2017-08-28 15:13:20,928 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 1) 2017-08-28 15:13:20,929 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:13:20,938 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,939 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933200, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:20,939 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,940 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,941 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:20,941 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,947 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,954 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:20,955 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,963 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:13:20,967 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 1) from Node3:1 2017-08-28 15:13:20,972 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,972 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:20,972 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 1) from Node4:1 2017-08-28 15:13:20,973 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 1) at 249497.121701001 2017-08-28 15:13:20,980 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:13:20,981 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:20,981 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:13:20,981 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:20,982 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 1) from Node1:1 2017-08-28 15:13:20,985 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:13:20,986 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,003 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node4 2017-08-28 15:13:21,004 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node4 2017-08-28 15:13:21,004 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node1 2017-08-28 15:13:21,004 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node1 2017-08-28 15:13:21,009 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node3 2017-08-28 15:13:21,011 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","instId":0,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","ppSeqNo":1,"ppTime":1503933200,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}' to Node3 2017-08-28 15:13:21,017 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:13:21,018 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:13:21,024 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,025 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:21,030 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,038 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,039 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,043 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-08-28 15:13:21,044 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-08-28 15:13:21,044 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,045 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,055 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 1) from Node1:1 2017-08-28 15:13:21,056 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:13:21,056 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,067 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:13:21,071 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:13:21,074 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}']}, 'Node4') 2017-08-28 15:13:21,074 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:13:21,075 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:21,077 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,077 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:21,078 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,078 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,079 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,079 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']} 2017-08-28 15:13:21,087 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']}, 'Node3') 2017-08-28 15:13:21,087 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']} 2017-08-28 15:13:21,088 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:21,088 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'op': 'PREPARE', 'ppTime': 1503933200, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:21,088 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:21,095 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,095 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:21,098 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,120 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 1) from Node4:0 2017-08-28 15:13:21,120 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 1) at 249497.269241496 2017-08-28 15:13:21,120 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:13:21,128 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,128 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:13:21,129 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,129 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 1) from Node3:0 2017-08-28 15:13:21,129 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'instId': 0, 'viewNo': 0, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppSeqNo': 1} 2017-08-28 15:13:21,132 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,134 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 1) from Node4:1 2017-08-28 15:13:21,135 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:13:21,139 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 1) 2017-08-28 15:13:21,140 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 1) 2017-08-28 15:13:21,140 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:13:21,140 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 1} 2017-08-28 15:13:21,141 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 1) 2017-08-28 15:13:21,147 | DEBUG | replica.py (1540) | _newCheckpointState | Node2:1 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,148 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,148 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} because already ordered 3 phase message 2017-08-28 15:13:21,165 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:13:21,170 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:13:21,171 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:13:21,180 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,180 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:21,180 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,191 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,193 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,193 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-08-28 15:13:21,196 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-08-28 15:13:21,196 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,202 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,209 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node3:0 2017-08-28 15:13:21,209 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:13:21,209 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,210 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:13:21,220 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,220 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,220 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,226 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node4:0 2017-08-28 15:13:21,238 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:13:21,241 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 1) 2017-08-28 15:13:21,242 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 1) 2017-08-28 15:13:21,242 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:13:21,242 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'ppTime': 1503933200, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'instId': 0, 'viewNo': 0, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'ppSeqNo': 1} 2017-08-28 15:13:21,247 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 1) 2017-08-28 15:13:21,248 | DEBUG | replica.py (1540) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,248 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,249 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:13:21,259 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,259 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:13:21,259 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-08-28 15:13:21,271 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 1 of 1 requests 2017-08-28 15:13:21,335 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:13:21,336 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to client 2017-08-28 15:13:21,336 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"rootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","txnTime":1503933200,"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368,"auditPath":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61","seqNo":10}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:21,342 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 1) for ledger 1 range (10, 10) 2017-08-28 15:13:21,343 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:21,347 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} because already ordered 3 phase message 2017-08-28 15:13:21,361 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.473 seconds 2017-08-28 15:13:25,189 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-08-28 15:13:25,191 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:25,192 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:25,192 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:13:25,192 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:25,202 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:25,203 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:25,208 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-08-28 15:13:30,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:30,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:30,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:30,303 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:13:35,218 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 11 2017-08-28 15:13:35,225 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:35,226 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:35,227 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:13:35,228 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:35,229 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:35,231 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:35,231 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-08-28 15:13:45,236 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 12 2017-08-28 15:13:45,236 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:45,237 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:45,237 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:13:45,237 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:45,238 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:45,238 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:45,238 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-08-28 15:13:45,308 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:45,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:13:45,315 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:45,316 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:13:55,243 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 13 2017-08-28 15:13:55,248 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:13:55,248 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:55,249 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:13:55,249 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:55,250 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:55,251 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:13:55,251 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-08-28 15:14:00,327 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:00,328 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:00,328 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:00,336 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:14:05,261 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 14 2017-08-28 15:14:05,274 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:05,274 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:05,274 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:05,275 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:05,276 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:05,279 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:05,281 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-08-28 15:14:15,084 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 9 2017-08-28 15:14:15,085 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:14:15,085 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:15,085 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 16 to run in 60 seconds 2017-08-28 15:14:15,094 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 2 2017-08-28 15:14:15,111 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:14:15,111 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-08-28 15:14:15,290 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 15 2017-08-28 15:14:15,295 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:15,296 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:15,297 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:15,303 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:15,303 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:15,304 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:15,304 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-08-28 15:14:15,346 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:15,364 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:15,364 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:15,365 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:14:25,312 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 17 2017-08-28 15:14:25,314 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:25,319 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:25,322 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:25,322 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:25,322 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:25,323 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:25,323 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-08-28 15:14:30,349 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:30,359 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:30,359 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:30,360 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:14:35,334 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 18 2017-08-28 15:14:35,335 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:35,335 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:35,336 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:35,336 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:35,336 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:35,337 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:35,345 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-08-28 15:14:45,350 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 19 2017-08-28 15:14:45,351 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:45,355 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:45,355 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:45,355 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:45,356 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:45,356 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:45,356 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-08-28 15:14:45,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:45,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:14:45,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:45,370 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:14:55,362 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 20 2017-08-28 15:14:55,372 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:14:55,372 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:55,373 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:14:55,373 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:55,373 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:55,374 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:14:55,374 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-08-28 15:15:00,375 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:00,376 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:00,376 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:00,377 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:15:05,388 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 21 2017-08-28 15:15:05,388 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:05,388 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:05,399 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:05,400 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:05,400 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:05,400 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:05,404 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-08-28 15:15:15,089 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 16 2017-08-28 15:15:15,090 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:15:15,090 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:15,091 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 23 to run in 60 seconds 2017-08-28 15:15:15,129 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 3 2017-08-28 15:15:15,134 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:15:15,135 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-08-28 15:15:15,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:15,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:15,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:15,389 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:15:15,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 22 2017-08-28 15:15:15,414 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:15,414 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:15,415 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:15,417 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:15,417 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:15,418 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:15,418 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-08-28 15:15:25,448 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 24 2017-08-28 15:15:25,455 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:25,465 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:25,465 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:25,465 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:25,466 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:25,466 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:25,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-08-28 15:15:30,418 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:30,419 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:30,419 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:30,425 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:15:35,471 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 25 2017-08-28 15:15:35,472 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:35,472 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:35,472 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:35,473 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:35,473 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:35,473 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:35,473 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-08-28 15:15:45,429 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:45,432 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:15:45,432 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:45,433 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:15:45,476 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 26 2017-08-28 15:15:45,477 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:45,477 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:45,478 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:45,478 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:45,478 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:45,478 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:45,480 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-08-28 15:15:55,488 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 27 2017-08-28 15:15:55,489 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:15:55,489 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:55,489 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:15:55,492 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:55,493 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:55,493 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:15:55,494 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-08-28 15:16:00,434 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:00,440 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:00,440 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:00,446 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:16:05,503 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 28 2017-08-28 15:16:05,504 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:05,513 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:05,513 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:05,513 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:05,514 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:05,514 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:05,514 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-08-28 15:16:15,102 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 23 2017-08-28 15:16:15,109 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:16:15,110 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:15,110 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 30 to run in 60 seconds 2017-08-28 15:16:15,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 4 2017-08-28 15:16:15,146 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:16:15,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-08-28 15:16:15,435 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:15,436 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:15,437 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:15,438 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:16:15,529 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 29 2017-08-28 15:16:15,531 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:15,535 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:15,535 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:15,535 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:15,538 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:15,539 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:15,540 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-08-28 15:16:25,549 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 31 2017-08-28 15:16:25,549 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:25,553 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:25,554 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:25,554 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:25,554 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:25,558 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:25,559 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-08-28 15:16:30,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:30,441 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:30,441 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:30,442 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:16:35,563 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 32 2017-08-28 15:16:35,564 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:35,575 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:35,577 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:35,578 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:35,578 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:35,579 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:35,580 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-08-28 15:16:45,477 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:45,484 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:16:45,485 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:45,485 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:16:45,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 33 2017-08-28 15:16:45,596 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:45,596 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:45,597 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:45,597 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:45,597 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:45,597 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:45,597 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-08-28 15:16:55,614 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 34 2017-08-28 15:16:55,627 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:16:55,636 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:55,637 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:16:55,637 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:55,637 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:55,637 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:16:55,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 35 to run in 10 seconds 2017-08-28 15:16:57,878 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:16:57,879 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:57,899 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:57,905 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:16:57,908 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:00,495 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:00,500 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:00,503 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:00,504 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:17:05,668 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 35 2017-08-28 15:17:05,669 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:05,669 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:05,679 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:17:05,683 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:05,684 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:05,684 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:05,684 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-08-28 15:17:07,414 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:07,420 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,421 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,422 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:07,434 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:12,684 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:12,684 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,693 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,695 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:12,716 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:15,121 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 30 2017-08-28 15:17:15,125 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:17:15,132 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:15,132 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 37 to run in 60 seconds 2017-08-28 15:17:15,154 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 5 2017-08-28 15:17:15,154 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:17:15,154 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-08-28 15:17:15,503 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:15,504 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:15,504 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:15,505 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:17:15,709 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 36 2017-08-28 15:17:15,715 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:15,716 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:15,716 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:17:15,719 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:15,720 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:15,726 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:15,727 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-08-28 15:17:22,326 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:22,333 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,333 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,352 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:22,353 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:25,742 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 38 2017-08-28 15:17:25,752 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:25,753 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:25,754 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:17:25,754 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:25,754 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:25,754 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:25,754 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-08-28 15:17:27,970 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:27,988 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:27,989 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:27,991 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:28,008 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:30,530 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:30,535 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:30,539 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:30,550 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:17:35,770 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 39 2017-08-28 15:17:35,782 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:35,783 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:35,783 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.6906639395854764 is acceptable. 2017-08-28 15:17:35,787 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:35,788 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:35,788 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:35,788 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-08-28 15:17:37,553 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:37,553 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,555 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,564 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:37,576 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:39,556 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:17:39,579 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933459424488 2017-08-28 15:17:39,580 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,581 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,581 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,581 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,581 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933459424488 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,582 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:39,582 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} to its replicas since not finalised 2017-08-28 15:17:39,596 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:39,610 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node4 2017-08-28 15:17:39,611 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node4 2017-08-28 15:17:39,623 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node1 2017-08-28 15:17:39,625 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node1 2017-08-28 15:17:39,626 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node3 2017-08-28 15:17:39,626 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933459424488,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"role":"2","type":"1","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"}}}' to Node3 2017-08-28 15:17:39,629 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:39,651 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,658 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,659 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:17:39,659 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,666 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,666 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,667 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to 0 replicas 2017-08-28 15:17:39,699 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 2 for ledger 1 with state root None 2017-08-28 15:17:39,711 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:39,712 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:17:39,712 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 2} 2017-08-28 15:17:39,712 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 2} with state root None 2017-08-28 15:17:39,713 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:39,727 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,728 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,728 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:39,728 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,736 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,736 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,736 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:17:39,737 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,737 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,737 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,737 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} to its replicas since already forwarded 2017-08-28 15:17:39,756 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933459424488, 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:39,757 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} 2017-08-28 15:17:39,757 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933459424488, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'role': '2', 'type': '1', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}} to its replicas since already forwarded 2017-08-28 15:17:39,779 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:39,799 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node4 2017-08-28 15:17:39,806 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node4 2017-08-28 15:17:39,807 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node1 2017-08-28 15:17:39,814 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node1 2017-08-28 15:17:39,815 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node3 2017-08-28 15:17:39,815 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933459,"discarded":1,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","ledgerId":1}' to Node3 2017-08-28 15:17:39,860 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.325 seconds 2017-08-28 15:17:39,872 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:39,886 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2} 2017-08-28 15:17:39,886 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:39,894 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2} 2017-08-28 15:17:39,920 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 2) from Node1:0 at 249756.068827967 2017-08-28 15:17:39,932 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2} is b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t', ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE 2017-08-28 15:17:39,960 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 2) at 249756.108819479 2017-08-28 15:17:39,960 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:17:39,965 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:39,965 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:39,965 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'discarded': 1, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2} with state root b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t' 2017-08-28 15:17:39,969 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 2) 2017-08-28 15:17:40,004 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:40,018 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:40,022 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,035 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:40,035 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,042 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,043 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,046 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,058 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node4 2017-08-28 15:17:40,068 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node4 2017-08-28 15:17:40,071 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node1 2017-08-28 15:17:40,076 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node1 2017-08-28 15:17:40,082 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node3 2017-08-28 15:17:40,083 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","instId":0,"viewNo":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","ppSeqNo":2,"ppTime":1503933459,"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8"}' to Node3 2017-08-28 15:17:40,101 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 2) from Node1:1 2017-08-28 15:17:40,101 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,102 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,102 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 2) from Node4:1 2017-08-28 15:17:40,113 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 2) at 249756.262207673 2017-08-28 15:17:40,113 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:17:40,120 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,120 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:17:40,130 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,148 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:40,149 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,149 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933459, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,155 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,156 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:40,163 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,164 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:40,189 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:40,200 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 2) from Node4:0 2017-08-28 15:17:40,200 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 2) at 249756.349347081 2017-08-28 15:17:40,210 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:17:40,216 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,217 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:17:40,217 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,225 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 2) from Node3:1 2017-08-28 15:17:40,225 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:17:40,226 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,234 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:40,236 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:40,250 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'op': 'PREPARE', 'ppTime': 1503933459, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,250 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:40,269 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,276 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,276 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-08-28 15:17:40,277 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-08-28 15:17:40,287 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,291 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,304 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:40,307 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 2) from Node3:0 2017-08-28 15:17:40,312 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'instId': 0, 'viewNo': 0, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppSeqNo': 2} 2017-08-28 15:17:40,323 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,325 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:40,336 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,337 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,337 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,344 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,344 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:40,352 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,352 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,353 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,355 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,378 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,385 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,390 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-08-28 15:17:40,390 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-08-28 15:17:40,390 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,396 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,397 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 2) from Node3:1 2017-08-28 15:17:40,409 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:17:40,409 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,424 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 2) from Node1:1 2017-08-28 15:17:40,425 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:17:40,431 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 2) 2017-08-28 15:17:40,432 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 2) 2017-08-28 15:17:40,432 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:17:40,443 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 2} 2017-08-28 15:17:40,444 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 2) 2017-08-28 15:17:40,454 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,455 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} because already ordered 3 phase message 2017-08-28 15:17:40,463 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:40,464 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,475 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,476 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,476 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,484 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,484 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,485 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,485 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:17:40,494 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-08-28 15:17:40,499 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:17:40,526 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node3:0 2017-08-28 15:17:40,529 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:17:40,531 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,540 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node4:0 2017-08-28 15:17:40,541 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:17:40,541 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 2) 2017-08-28 15:17:40,541 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 2) 2017-08-28 15:17:40,542 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:17:40,548 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'ppTime': 1503933459, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'instId': 0, 'viewNo': 0, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'ppSeqNo': 2} 2017-08-28 15:17:40,548 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 2) 2017-08-28 15:17:40,549 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,564 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} because already ordered 3 phase message 2017-08-28 15:17:40,581 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 2 of 1 requests 2017-08-28 15:17:40,596 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:40,606 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to client 2017-08-28 15:17:40,607 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1503933459,"rootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","role":"2","signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","verkey":"~RmCt3RtDvKDfQBSKxo4qvy","reqId":1503933459424488,"auditPath":["A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":11,"dest":"XhYtvJqezMUKfF6KVNaGmT"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:40,618 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 2) for ledger 1 range (11, 11) 2017-08-28 15:17:40,619 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:40,645 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.774 seconds 2017-08-28 15:17:42,730 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:42,731 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,740 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,742 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:42,742 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:45,559 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:45,560 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:17:45,562 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:45,563 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:17:45,805 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 40 2017-08-28 15:17:45,808 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:45,808 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:45,810 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.8236152811863385 is acceptable. 2017-08-28 15:17:45,811 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:45,813 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:45,816 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:45,816 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-08-28 15:17:46,187 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:17:46,205 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933466149129 2017-08-28 15:17:46,212 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,218 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,219 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,223 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,223 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933466149129 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,228 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:46,237 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} to its replicas since not finalised 2017-08-28 15:17:46,237 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:46,243 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node4 2017-08-28 15:17:46,255 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node4 2017-08-28 15:17:46,261 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node1 2017-08-28 15:17:46,262 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node1 2017-08-28 15:17:46,262 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node3 2017-08-28 15:17:46,271 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933466149129,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"role":"2","type":"1","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"}}}' to Node3 2017-08-28 15:17:46,282 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:46,289 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,295 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,300 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:17:46,300 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,301 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,311 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,312 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:46,312 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,324 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,324 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,324 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:17:46,331 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,332 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,335 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,345 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to 0 replicas 2017-08-28 15:17:46,345 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,346 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,348 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} to its replicas since already forwarded 2017-08-28 15:17:46,349 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933466149129, 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:46,362 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} 2017-08-28 15:17:46,362 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933466149129, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'role': '2', 'type': '1', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}} to its replicas since already forwarded 2017-08-28 15:17:46,385 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 3 for ledger 1 with state root None 2017-08-28 15:17:46,386 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:46,391 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:17:46,392 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 3} 2017-08-28 15:17:46,392 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 3} with state root None 2017-08-28 15:17:46,420 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:46,434 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node4 2017-08-28 15:17:46,434 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node4 2017-08-28 15:17:46,449 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node1 2017-08-28 15:17:46,449 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node1 2017-08-28 15:17:46,450 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node3 2017-08-28 15:17:46,450 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933466,"discarded":1,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","ledgerId":1}' to Node3 2017-08-28 15:17:46,472 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:46,473 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3} 2017-08-28 15:17:46,473 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:46,483 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3} 2017-08-28 15:17:46,499 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 3) from Node1:0 at 249762.64780097 2017-08-28 15:17:46,515 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3} is b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`', 32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb 2017-08-28 15:17:46,517 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 3) at 249762.666407882 2017-08-28 15:17:46,528 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:17:46,528 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,534 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,534 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'discarded': 1, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3} with state root b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`' 2017-08-28 15:17:46,535 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 3) 2017-08-28 15:17:46,539 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:46,588 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node4 2017-08-28 15:17:46,589 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node4 2017-08-28 15:17:46,593 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node1 2017-08-28 15:17:46,593 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node1 2017-08-28 15:17:46,594 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node3 2017-08-28 15:17:46,604 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","instId":0,"viewNo":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","ppSeqNo":3,"ppTime":1503933466,"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314"}' to Node3 2017-08-28 15:17:46,611 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:46,621 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,621 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:46,621 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,622 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,633 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,634 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,660 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 3) from Node1:1 2017-08-28 15:17:46,660 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,661 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,669 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 3) from Node4:1 2017-08-28 15:17:46,669 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 3) at 249762.818129336 2017-08-28 15:17:46,669 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:17:46,670 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,680 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:17:46,681 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,682 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:46,698 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,706 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933466, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:46,707 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,715 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,716 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,716 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,740 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:46,740 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 3) from Node4:0 2017-08-28 15:17:46,751 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 3) at 249762.900624767 2017-08-28 15:17:46,752 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:17:46,752 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,761 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:17:46,762 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,762 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 3) from Node3:1 2017-08-28 15:17:46,772 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:17:46,772 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,795 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,796 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,796 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-08-28 15:17:46,797 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-08-28 15:17:46,807 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,811 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,821 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:46,836 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:46,843 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,844 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'op': 'PREPARE', 'ppTime': 1503933466, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:46,856 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,856 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,856 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:46,857 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,876 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,877 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,883 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-08-28 15:17:46,891 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-08-28 15:17:46,895 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,897 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,909 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 3) from Node3:0 2017-08-28 15:17:46,920 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'instId': 0, 'viewNo': 0, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppSeqNo': 3} 2017-08-28 15:17:46,921 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,921 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 3) from Node1:1 2017-08-28 15:17:46,928 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:17:46,928 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:46,929 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:46,944 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:46,958 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}']}, 'Node4') 2017-08-28 15:17:46,958 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:46,959 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,959 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,968 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,969 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,976 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,976 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,979 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:17:46,987 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']}, 'Node3') 2017-08-28 15:17:46,987 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:17:46,991 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,995 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:46,996 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:46,996 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:47,001 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:47,008 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:47,008 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:47,009 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:17:47,021 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-08-28 15:17:47,044 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node4:0 2017-08-28 15:17:47,055 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:17:47,056 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,059 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node3:0 2017-08-28 15:17:47,060 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:17:47,071 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 3) 2017-08-28 15:17:47,071 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 3) 2017-08-28 15:17:47,072 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:17:47,072 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 0, 'viewNo': 0, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'ppSeqNo': 3} 2017-08-28 15:17:47,082 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 3) 2017-08-28 15:17:47,082 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,091 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} because already ordered 3 phase message 2017-08-28 15:17:47,093 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 3) from Node4:1 2017-08-28 15:17:47,093 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:17:47,095 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 3) 2017-08-28 15:17:47,100 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 3) 2017-08-28 15:17:47,101 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:17:47,101 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933466, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 3} 2017-08-28 15:17:47,101 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 3) 2017-08-28 15:17:47,102 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,103 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} because already ordered 3 phase message 2017-08-28 15:17:47,119 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 3 of 1 requests 2017-08-28 15:17:47,130 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:47,133 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to client 2017-08-28 15:17:47,152 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1503933466,"rootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","role":"2","signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","verkey":"~2a4KepmnVVAPGo5gNA3P3p","reqId":1503933466149129,"auditPath":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":12,"dest":"FBQ5W7LxRp8bLeVNEafjpd"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:47,174 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 3) for ledger 1 range (12, 12) 2017-08-28 15:17:47,174 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:17:47,175 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:47,237 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.052 seconds 2017-08-28 15:17:51,708 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:17:51,710 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933471684615 2017-08-28 15:17:51,739 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,740 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,744 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,745 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,745 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933471684615 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,756 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:51,757 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} to its replicas since not finalised 2017-08-28 15:17:51,764 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:51,767 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node4 2017-08-28 15:17:51,779 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node4 2017-08-28 15:17:51,780 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node1 2017-08-28 15:17:51,780 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node1 2017-08-28 15:17:51,795 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node3 2017-08-28 15:17:51,799 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933471684615,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"role":"100","type":"1","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"}}}' to Node3 2017-08-28 15:17:51,808 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:51,823 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,824 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,824 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:51,825 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,837 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,837 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,837 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:17:51,839 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,853 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,853 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,854 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:17:51,860 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,860 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,867 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,868 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to 0 replicas 2017-08-28 15:17:51,868 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,880 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,880 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} to its replicas since already forwarded 2017-08-28 15:17:51,880 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933471684615, 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:51,891 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} 2017-08-28 15:17:51,896 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933471684615, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'role': '100', 'type': '1', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}} to its replicas since already forwarded 2017-08-28 15:17:51,899 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 4 for ledger 1 with state root None 2017-08-28 15:17:51,918 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:51,924 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:17:51,924 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 4} 2017-08-28 15:17:51,924 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 4} with state root None 2017-08-28 15:17:51,946 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 4} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:51,967 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node4 2017-08-28 15:17:51,968 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node4 2017-08-28 15:17:51,970 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node1 2017-08-28 15:17:51,973 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node1 2017-08-28 15:17:51,973 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node3 2017-08-28 15:17:51,974 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"txnRootHash":null,"ppSeqNo":4,"ppTime":1503933471,"discarded":1,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","ledgerId":1}' to Node3 2017-08-28 15:17:52,005 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.306 seconds 2017-08-28 15:17:52,049 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:52,050 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4} 2017-08-28 15:17:52,060 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,061 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4} 2017-08-28 15:17:52,083 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 4) from Node1:0 at 249768.231742615 2017-08-28 15:17:52,085 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4} is b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3', 8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg 2017-08-28 15:17:52,108 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 4) at 249768.256755487 2017-08-28 15:17:52,108 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:17:52,108 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,109 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,117 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'discarded': 1, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4} with state root b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3' 2017-08-28 15:17:52,118 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 4) 2017-08-28 15:17:52,131 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:52,141 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,142 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,142 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,164 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:52,165 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 4) from Node1:1 2017-08-28 15:17:52,172 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,172 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,183 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:52,188 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,188 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933471, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,189 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,200 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,200 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,202 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,223 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node4 2017-08-28 15:17:52,235 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node4 2017-08-28 15:17:52,236 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node1 2017-08-28 15:17:52,236 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node1 2017-08-28 15:17:52,237 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node3 2017-08-28 15:17:52,250 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","instId":0,"viewNo":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","ppSeqNo":4,"ppTime":1503933471,"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a"}' to Node3 2017-08-28 15:17:52,251 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 4) from Node3:1 2017-08-28 15:17:52,265 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 4) at 249768.414317933 2017-08-28 15:17:52,266 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:17:52,266 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,272 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1 2017-08-28 15:17:52,279 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,280 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 4) from Node4:1 2017-08-28 15:17:52,280 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': None, 'ppSeqNo': 4} 2017-08-28 15:17:52,281 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,291 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:52,296 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,307 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'op': 'PREPARE', 'ppTime': 1503933471, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,308 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,315 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,316 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,316 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,316 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,316 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,334 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:52,342 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:52,354 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:52,355 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 4) from Node3:0 2017-08-28 15:17:52,364 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 4) at 249768.513149264 2017-08-28 15:17:52,365 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:17:52,365 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,378 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1 2017-08-28 15:17:52,378 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,385 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 4) from Node4:0 2017-08-28 15:17:52,386 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'instId': 0, 'viewNo': 0, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppSeqNo': 4} 2017-08-28 15:17:52,387 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,396 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:52,406 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,406 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,407 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,407 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,421 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,421 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,429 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,430 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,436 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,449 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,464 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,467 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node1 2017-08-28 15:17:52,468 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node1 2017-08-28 15:17:52,468 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,468 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,491 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:52,494 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 4) from Node1:1 2017-08-28 15:17:52,494 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1 2017-08-28 15:17:52,495 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,495 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 4) from Node3:1 2017-08-28 15:17:52,514 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:17:52,514 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 4) 2017-08-28 15:17:52,519 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 4) 2017-08-28 15:17:52,522 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:17:52,522 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 4} 2017-08-28 15:17:52,523 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 4) 2017-08-28 15:17:52,527 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,528 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} because already ordered 3 phase message 2017-08-28 15:17:52,541 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:52,542 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,556 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,556 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,556 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,557 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,557 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,583 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,584 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,584 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node1 2017-08-28 15:17:52,597 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node1 2017-08-28 15:17:52,597 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,598 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,615 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:17:52,620 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 4) from Node3:0 2017-08-28 15:17:52,620 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1 2017-08-28 15:17:52,630 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,631 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 4) from Node4:0 2017-08-28 15:17:52,635 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:17:52,636 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 4) 2017-08-28 15:17:52,636 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 4) 2017-08-28 15:17:52,637 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:17:52,644 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'ppTime': 1503933471, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'instId': 0, 'viewNo': 0, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'ppSeqNo': 4} 2017-08-28 15:17:52,644 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 4) 2017-08-28 15:17:52,644 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,646 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:52,665 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,666 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 4}, 'Node1') 2017-08-28 15:17:52,671 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} 2017-08-28 15:17:52,694 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 4 of 1 requests 2017-08-28 15:17:52,695 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:52,700 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to client 2017-08-28 15:17:52,700 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1503933471,"rootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","role":"100","signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","verkey":"~YHDYsnaN59TzUsLqvPYdRr","reqId":1503933471684615,"auditPath":["4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":13,"dest":"6XiqM9DwYDgFnmyQQBFr72"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:52,718 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 4) for ledger 1 range (13, 13) 2017-08-28 15:17:52,718 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:52,718 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 4} because already ordered 3 phase message 2017-08-28 15:17:52,762 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.717 seconds 2017-08-28 15:17:55,823 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 41 2017-08-28 15:17:55,840 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:17:55,841 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:55,841 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.8858553382264499 is acceptable. 2017-08-28 15:17:55,842 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:55,843 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:55,856 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:17:55,856 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-08-28 15:17:57,611 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:57,616 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil': INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:17:57,616 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 1, 'reason': 26}, b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil') 2017-08-28 15:17:57,623 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:17:57,624 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:57,625 | DEBUG | node.py (1939) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-08-28 15:17:57,632 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.8858553382264499 is acceptable. 2017-08-28 15:17:57,632 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:57,632 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:57,633 | DEBUG | node.py (1927) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:17:58,108 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:58,118 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:58,118 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:58,133 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:58,138 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:58,947 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:17:58,958 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933478716086 2017-08-28 15:17:58,959 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:58,966 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:58,967 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,974 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:58,981 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933478716086 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,982 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:58,989 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} to its replicas since not finalised 2017-08-28 15:17:58,989 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:59,002 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node4 2017-08-28 15:17:59,003 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node4 2017-08-28 15:17:59,010 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node1 2017-08-28 15:17:59,011 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node1 2017-08-28 15:17:59,018 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node3 2017-08-28 15:17:59,019 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933478716086,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"role":"101","type":"1","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"}}}' to Node3 2017-08-28 15:17:59,025 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:59,038 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,047 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,048 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:17:59,048 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,058 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,059 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,066 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:17:59,066 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,067 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,070 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,077 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:17:59,077 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,078 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,090 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:59,090 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to 0 replicas 2017-08-28 15:17:59,090 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,097 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:59,097 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} to its replicas since already forwarded 2017-08-28 15:17:59,097 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933478716086, 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:17:59,110 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} 2017-08-28 15:17:59,111 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933478716086, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'role': '101', 'type': '1', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}} to its replicas since already forwarded 2017-08-28 15:17:59,119 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 5 for ledger 1 with state root None 2017-08-28 15:17:59,133 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:59,133 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:17:59,140 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 5} 2017-08-28 15:17:59,140 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 5} with state root None 2017-08-28 15:17:59,164 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 5} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:59,183 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node4 2017-08-28 15:17:59,191 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node4 2017-08-28 15:17:59,195 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node1 2017-08-28 15:17:59,195 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node1 2017-08-28 15:17:59,196 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node3 2017-08-28 15:17:59,196 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"txnRootHash":null,"ppSeqNo":5,"ppTime":1503933479,"discarded":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ledgerId":1}' to Node3 2017-08-28 15:17:59,219 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:59,220 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5} 2017-08-28 15:17:59,231 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,232 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5} 2017-08-28 15:17:59,239 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 5) from Node1:0 at 249775.387716977 2017-08-28 15:17:59,251 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5} is b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6', CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA 2017-08-28 15:17:59,265 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 5) at 249775.413705764 2017-08-28 15:17:59,275 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:17:59,276 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,276 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,284 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'discarded': 1, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5} with state root b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6' 2017-08-28 15:17:59,285 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 5) 2017-08-28 15:17:59,310 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:59,332 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:59,336 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,341 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,342 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,367 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node4 2017-08-28 15:17:59,368 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node4 2017-08-28 15:17:59,368 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node1 2017-08-28 15:17:59,369 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node1 2017-08-28 15:17:59,377 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node3 2017-08-28 15:17:59,378 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","instId":0,"viewNo":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","ppSeqNo":5,"ppTime":1503933479,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210"}' to Node3 2017-08-28 15:17:59,380 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 5) from Node1:1 2017-08-28 15:17:59,399 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,401 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,403 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:17:59,419 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,420 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,420 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,448 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 5) from Node4:1 2017-08-28 15:17:59,455 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 5) at 249775.60466952 2017-08-28 15:17:59,456 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:17:59,456 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,456 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} commits where f is 1 2017-08-28 15:17:59,462 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,474 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:59,483 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']} 2017-08-28 15:17:59,488 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']}, 'Node3') 2017-08-28 15:17:59,488 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']} 2017-08-28 15:17:59,499 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,503 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'op': 'PREPARE', 'ppTime': 1503933479, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,504 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,504 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,505 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933479, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,508 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,520 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,520 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,520 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,552 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:59,555 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 5) from Node3:0 2017-08-28 15:17:59,560 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 5) at 249775.709205292 2017-08-28 15:17:59,561 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:17:59,561 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,561 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} commits where f is 1 2017-08-28 15:17:59,571 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,571 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 5) from Node4:0 2017-08-28 15:17:59,576 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'instId': 0, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppSeqNo': 5} 2017-08-28 15:17:59,579 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,584 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 5) from Node3:1 2017-08-28 15:17:59,584 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'txnRootHash': None, 'ppSeqNo': 5} 2017-08-28 15:17:59,595 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,609 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,610 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,621 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node1 2017-08-28 15:17:59,621 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node1 2017-08-28 15:17:59,622 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,622 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,640 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:17:59,642 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:17:59,655 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:59,655 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}']}, 'Node4') 2017-08-28 15:17:59,664 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:59,668 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,668 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,673 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,674 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,683 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,683 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,687 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,688 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,688 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,689 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,689 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,689 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,722 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,723 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,723 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node1 2017-08-28 15:17:59,731 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node1 2017-08-28 15:17:59,731 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,732 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,742 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 5) from Node4:0 2017-08-28 15:17:59,751 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} commits where f is 1 2017-08-28 15:17:59,751 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,759 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 5) from Node4:1 2017-08-28 15:17:59,760 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} commits where f is 1 2017-08-28 15:17:59,765 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,765 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 5) from Node3:1 2017-08-28 15:17:59,766 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:17:59,766 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 5) 2017-08-28 15:17:59,767 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 5) 2017-08-28 15:17:59,777 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:17:59,778 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 5} 2017-08-28 15:17:59,778 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 5) 2017-08-28 15:17:59,783 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,789 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 5} because already ordered 3 phase message 2017-08-28 15:17:59,790 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:17:59,798 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,798 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,798 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,815 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,816 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 5}, 'Node1') 2017-08-28 15:17:59,816 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} 2017-08-28 15:17:59,843 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:17:59,850 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 5) from Node3:0 2017-08-28 15:17:59,851 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:17:59,856 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 5) 2017-08-28 15:17:59,860 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 5) 2017-08-28 15:17:59,861 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:17:59,862 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'ppTime': 1503933479, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'instId': 0, 'viewNo': 0, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'ppSeqNo': 5} 2017-08-28 15:17:59,862 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 5) 2017-08-28 15:17:59,878 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,878 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 5} because already ordered 3 phase message 2017-08-28 15:17:59,905 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 5 of 1 requests 2017-08-28 15:17:59,918 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:59,930 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to client 2017-08-28 15:17:59,931 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1503933479,"rootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","role":"101","signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","verkey":"~RTJS62PRHurqHJnbnT1Hgi","reqId":1503933478716086,"auditPath":["HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":14,"dest":"Rhx2qwaeiGyhU9vn4ynHSS"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:59,938 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 5) for ledger 1 range (14, 14) 2017-08-28 15:17:59,938 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:00,002 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.069 seconds 2017-08-28 15:18:00,575 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:00,576 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:00,580 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:00,599 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:18:04,224 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:18:04,238 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933484172953 2017-08-28 15:18:04,247 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,248 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,249 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,249 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,259 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933484172953 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,259 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:04,260 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} to its replicas since not finalised 2017-08-28 15:18:04,260 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:04,276 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node4 2017-08-28 15:18:04,277 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node4 2017-08-28 15:18:04,285 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node1 2017-08-28 15:18:04,285 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node1 2017-08-28 15:18:04,295 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node3 2017-08-28 15:18:04,296 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933484172953,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"}}}' to Node3 2017-08-28 15:18:04,297 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:04,319 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,320 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,327 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:18:04,328 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,329 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,337 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,338 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:18:04,349 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,350 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,351 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,355 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:04,356 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,356 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,371 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,371 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to 0 replicas 2017-08-28 15:18:04,372 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,379 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,387 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} to its replicas since already forwarded 2017-08-28 15:18:04,388 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933484172953, 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:04,391 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} 2017-08-28 15:18:04,392 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933484172953, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}} to its replicas since already forwarded 2017-08-28 15:18:04,394 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:1 creating batch 6 for ledger 1 with state root None 2017-08-28 15:18:04,416 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:18:04,416 | DEBUG | replica.py (2133) | send | Node2:1 sending PrePrepare 2017-08-28 15:18:04,416 | TRACE | replica.py (2134) | send | Node2:1 sending PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 6} 2017-08-28 15:18:04,418 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 6} with state root None 2017-08-28 15:18:04,448 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 6} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:04,471 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node4 2017-08-28 15:18:04,472 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node4 2017-08-28 15:18:04,472 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node1 2017-08-28 15:18:04,479 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node1 2017-08-28 15:18:04,483 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node3 2017-08-28 15:18:04,484 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":0,"stateRootHash":null,"instId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"txnRootHash":null,"ppSeqNo":6,"ppTime":1503933484,"discarded":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ledgerId":1}' to Node3 2017-08-28 15:18:04,487 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:04,504 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6} 2017-08-28 15:18:04,505 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPREPARE', 'viewNo': 0, 'discarded': 1, 'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:04,511 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6} 2017-08-28 15:18:04,532 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 6) from Node1:0 at 249780.680985227 2017-08-28 15:18:04,542 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6} is b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>', Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn 2017-08-28 15:18:04,549 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 6) at 249780.697982173 2017-08-28 15:18:04,554 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:18:04,554 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,554 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,555 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'discarded': 1, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6} with state root b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>' 2017-08-28 15:18:04,569 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 6) 2017-08-28 15:18:04,591 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:04,617 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node4 2017-08-28 15:18:04,617 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node4 2017-08-28 15:18:04,619 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node1 2017-08-28 15:18:04,620 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node1 2017-08-28 15:18:04,628 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node3 2017-08-28 15:18:04,628 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","instId":0,"viewNo":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"ppTime":1503933484,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337"}' to Node3 2017-08-28 15:18:04,647 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:04,648 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,648 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:04,654 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,678 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 6) from Node1:1 2017-08-28 15:18:04,678 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,684 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,686 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:04,699 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,699 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,703 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,717 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 6) from Node4:1 2017-08-28 15:18:04,730 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 6) at 249780.879294968 2017-08-28 15:18:04,730 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:18:04,731 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,731 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} commits where f is 1 2017-08-28 15:18:04,736 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,737 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:04,750 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']} 2017-08-28 15:18:04,758 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']}, 'Node3') 2017-08-28 15:18:04,759 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']} 2017-08-28 15:18:04,767 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,768 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'op': 'PREPARE', 'ppTime': 1503933484, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,768 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,777 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,778 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933484, 'txnRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,778 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,790 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:04,815 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 6) from Node3:0 2017-08-28 15:18:04,815 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 6) at 249780.964183212 2017-08-28 15:18:04,819 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:18:04,820 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,820 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} commits where f is 1 2017-08-28 15:18:04,820 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,820 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(0, 6) from Node3:1 2017-08-28 15:18:04,820 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'instId': 1, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': None, 'ppSeqNo': 6} 2017-08-28 15:18:04,820 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,846 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:04,846 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,847 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,848 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,869 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:04,870 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:04,875 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node1 2017-08-28 15:18:04,876 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node1 2017-08-28 15:18:04,886 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:04,887 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:04,893 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:04,905 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(0, 6) from Node4:0 2017-08-28 15:18:04,906 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'instId': 0, 'viewNo': 0, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppSeqNo': 6} 2017-08-28 15:18:04,906 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,914 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:04,915 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:18:04,924 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}']}, 'Node4') 2017-08-28 15:18:04,930 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:18:04,930 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,931 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,941 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,941 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,941 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,942 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,943 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']} 2017-08-28 15:18:04,944 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']}, 'Node3') 2017-08-28 15:18:04,953 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']} 2017-08-28 15:18:04,953 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,953 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,954 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,958 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,958 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,958 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,969 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,969 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:04,969 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:04,999 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:05,000 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:05,007 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node1 2017-08-28 15:18:05,008 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node1 2017-08-28 15:18:05,008 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:05,019 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:05,033 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 6) from Node4:0 2017-08-28 15:18:05,034 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} commits where f is 1 2017-08-28 15:18:05,034 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,035 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 6) from Node3:0 2017-08-28 15:18:05,035 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:18:05,047 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (0, 6) 2017-08-28 15:18:05,047 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (0, 6) 2017-08-28 15:18:05,047 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:18:05,055 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 0, 'viewNo': 0, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'ppSeqNo': 6} 2017-08-28 15:18:05,059 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (0, 6) 2017-08-28 15:18:05,060 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,060 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 6) from Node4:1 2017-08-28 15:18:05,067 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} commits where f is 1 2017-08-28 15:18:05,068 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,068 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 6) from Node3:1 2017-08-28 15:18:05,077 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:18:05,077 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (0, 6) 2017-08-28 15:18:05,077 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (0, 6) 2017-08-28 15:18:05,078 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:18:05,086 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933484, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'instId': 1, 'viewNo': 0, 'txnRootHash': None, 'ledgerId': 1, 'ppSeqNo': 6} 2017-08-28 15:18:05,087 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (0, 6) 2017-08-28 15:18:05,088 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,099 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 6} because already ordered 3 phase message 2017-08-28 15:18:05,100 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:05,101 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:05,114 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 6}, 'Node1') 2017-08-28 15:18:05,115 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} 2017-08-28 15:18:05,143 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 0 6 of 1 requests 2017-08-28 15:18:05,145 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:18:05,146 | DEBUG | node.py (2344) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to client 2017-08-28 15:18:05,146 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"rootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","txnTime":1503933484,"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","verkey":"~UWK9xwiKsqMQSBmymYpR5v","reqId":1503933484172953,"auditPath":["ijqq5QRv5GQwp9riPYGkR5Xkz9QxMds39Wc21R4V9Pt","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","seqNo":15,"dest":"5qhSKMkpqui6taGLTecwqf"}}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:05,163 | DEBUG | node.py (2263) | executeBatch | Node2 storing 3PC key (0, 6) for ledger 1 range (15, 15) 2017-08-28 15:18:05,164 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:18:05,164 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:05,164 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 6} because already ordered 3 phase message 2017-08-28 15:18:05,235 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.039 seconds 2017-08-28 15:18:05,859 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 42 2017-08-28 15:18:05,868 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:05,869 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:05,869 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 0.9064801276753965 is acceptable. 2017-08-28 15:18:05,869 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:05,885 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:05,885 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:05,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 43 to run in 10 seconds 2017-08-28 15:18:07,167 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:07,177 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7': INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:18:07,178 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 1, 'reason': 26}, b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7') 2017-08-28 15:18:07,182 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-08-28 15:18:07,190 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,198 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 1 from 0 2017-08-28 15:18:07,198 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action _check_view_change_completed with id 44 to run in 60 seconds 2017-08-28 15:18:07,205 | DEBUG | replica.py ( 453) | on_view_change_start | Node2:0 setting last prepared for master to (0, 6) 2017-08-28 15:18:07,206 | DEBUG | node.py (2127) | startViewChange | Node2 resetting monitor stats after view change 2017-08-28 15:18:07,206 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:18:07,213 | DEBUG | replica.py ( 406) | primaryName | Node2:0 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,213 | DEBUG | replica.py ( 406) | primaryName | Node2:1 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,294 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node2 changed to view 1, will start catchup now 2017-08-28 15:18:07,295 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,295 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,296 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,296 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,296 | DEBUG | node.py (2173) | start_catchup | Node2 reverted 0 batches before starting catch up 2017-08-28 15:18:07,296 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,297 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 0 2017-08-28 15:18:07,297 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:07,297 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 0 2017-08-28 15:18:07,299 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,311 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 0 2017-08-28 15:18:07,312 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,312 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 0 2017-08-28 15:18:07,312 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:07,317 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:07,317 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:07,318 | DEBUG | node.py (1927) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:18:07,349 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,349 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,356 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,356 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node4 2017-08-28 15:18:07,357 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,357 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,365 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,365 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-08-28 15:18:07,374 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,374 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,375 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,380 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-08-28 15:18:07,395 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:07,396 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,405 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:18:07,405 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,406 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,406 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,415 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,416 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,416 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,416 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,426 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:07,426 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:18:07,427 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:07,440 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,440 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,440 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,441 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,443 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,451 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,452 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,461 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,461 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,461 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,467 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:18:07,467 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,471 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,472 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:18:07,495 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,496 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,497 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,508 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,514 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,526 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,537 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,539 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,550 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,550 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,552 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node4 2017-08-28 15:18:07,566 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,566 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,576 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,578 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-08-28 15:18:07,612 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,612 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,613 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:18:07,626 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-08-28 15:18:07,646 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 4 messages through listener 2017-08-28 15:18:07,646 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:07,656 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:18:07,657 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:07,672 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,672 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,672 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,679 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,680 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,680 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,680 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,693 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,693 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,696 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,704 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:18:07,704 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,704 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,704 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:18:07,712 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,712 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,712 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,713 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:18:07,718 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']}, 'Node3') 2017-08-28 15:18:07,725 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:18:07,726 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,726 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,726 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,736 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,736 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,744 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,744 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node4 2017-08-28 15:18:07,751 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,751 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,766 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node4 2017-08-28 15:18:07,767 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,774 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node1 2017-08-28 15:18:07,781 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,787 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,787 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node1'} that its ledger of type 0 is latest 2017-08-28 15:18:07,794 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:18:07,795 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:18:07,795 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,795 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,800 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,800 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,800 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:18:07,800 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:18:07,809 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:18:07,809 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,810 | DEBUG | node.py ( 923) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node4', 'Node1', 'Node3'} 2017-08-28 15:18:07,817 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:18:07,817 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-08-28 15:18:07,818 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:07,824 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-08-28 15:18:07,825 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,831 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-08-28 15:18:07,832 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,832 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-08-28 15:18:07,832 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:18:07,833 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:18:07,843 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node1 2017-08-28 15:18:07,843 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,848 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,854 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node3 2017-08-28 15:18:07,854 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,862 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,862 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:18:07,868 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node3 2017-08-28 15:18:07,868 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,876 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,876 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:18:07,878 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,890 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,898 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:07,898 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-08-28 15:18:07,899 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:07,906 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,907 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:07,912 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:18:07,912 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,913 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,924 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:07,924 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:18:07,925 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:07,932 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:07,934 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:07,949 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:07,949 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:07,949 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:07,958 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,959 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:18:07,963 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,969 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,969 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:18:07,969 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,970 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,978 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:18:07,978 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,987 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,987 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:07,988 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,995 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,996 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:07,996 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,002 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,003 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:18:08,012 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,013 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,013 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,018 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,018 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,019 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,025 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,026 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,037 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,038 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,045 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,048 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,048 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,062 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,078 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,079 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:08,085 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-08-28 15:18:08,092 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,093 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,100 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:08,110 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:18:08,111 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,111 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,117 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:18:08,119 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:18:08,129 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:08,141 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,144 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:18:08,154 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,155 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,163 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:08,163 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,164 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,167 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:08,168 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,176 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:18:08,181 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-08-28 15:18:08,182 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:18:08,193 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,193 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,201 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,201 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,201 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,207 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,208 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:08,219 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,220 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:08,230 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,230 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:18:08,230 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,241 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,247 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:18:08,248 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,252 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node4 2017-08-28 15:18:08,253 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,262 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,269 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node4 2017-08-28 15:18:08,270 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,270 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,276 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:18:08,287 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,288 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,292 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:18:08,292 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:18:08,293 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,298 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,299 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,299 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,304 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,304 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:18:08,305 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:18:08,305 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:18:08,316 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,316 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:18:08,317 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-08-28 15:18:08,326 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,326 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-08-28 15:18:08,327 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:08,333 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-08-28 15:18:08,333 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,334 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-08-28 15:18:08,343 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,344 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-08-28 15:18:08,351 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:18:08,351 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-08-28 15:18:08,356 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:18:08,356 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:18:08,365 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,366 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,381 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:18:08,382 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,388 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,389 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:18:08,389 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:18:08,396 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,397 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,397 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:18:08,415 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,415 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,428 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,428 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:18:08,429 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,436 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,436 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,436 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:18:08,437 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,448 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,448 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,457 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:18:08,472 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:08,472 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,474 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,484 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,487 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,495 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:18:08,496 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,496 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,496 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:18:08,497 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,503 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,509 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:18:08,510 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,510 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,511 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,520 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,520 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,520 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,527 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,537 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,538 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:18:08,545 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,546 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,553 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:18:08,566 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,567 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,576 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:18:08,588 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,588 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,588 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,590 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:18:08,616 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:08,617 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:08,619 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:18:08,627 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:08,628 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,628 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,631 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,639 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,641 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,641 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,642 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,654 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:18:08,658 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,664 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,665 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:18:08,665 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,666 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,676 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:18:08,677 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,678 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,684 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,697 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node1 2017-08-28 15:18:08,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,699 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node1 2017-08-28 15:18:08,709 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,709 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,711 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,723 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,724 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:18:08,734 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:18:08,748 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:08,758 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:18:08,759 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']}, 'Node3') 2017-08-28 15:18:08,764 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:18:08,764 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,775 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,776 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,776 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,783 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,785 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,785 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,786 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:18:08,786 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,798 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,798 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,803 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,804 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,813 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,813 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,814 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node3 2017-08-28 15:18:08,826 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,826 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,827 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:18:08,827 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} that it does not need catchup 2017-08-28 15:18:08,833 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,833 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,834 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,834 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,840 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,842 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:18:08,842 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:18:08,842 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:18:08,853 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,853 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:18:08,854 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,861 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:18:08,861 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:18:08,862 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 3 ViewChangeDone messages 2017-08-28 15:18:08,862 | DEBUG | node.py (1550) | caught_up_for_current_view | Node2 does not have view change quorum for view 1 2017-08-28 15:18:08,870 | DEBUG | node.py (1534) | is_catchup_needed | Node2 is not caught up for the current view 1 2017-08-28 15:18:08,870 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,870 | DEBUG | node.py (1539) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-08-28 15:18:08,870 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-08-28 15:18:08,880 | DEBUG | primary_decider.py ( 131) | send | Node2's elector sending VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} 2017-08-28 15:18:08,880 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-08-28 15:18:08,880 | DEBUG | primary_selector.py ( 265) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:08,881 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node3 2017-08-28 15:18:08,891 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,901 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,902 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node4 2017-08-28 15:18:08,903 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,910 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,911 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:18:08,915 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node4 2017-08-28 15:18:08,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,926 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:18:08,934 | DEBUG | node.py (2566) | send | Node2 sending message VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:08,944 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:18:08,945 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:18:08,949 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:18:08,949 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:18:08,949 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:18:08,950 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:18:08,966 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:08,974 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:08,982 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 1, 'name': 'Node2'}, 'Node4') 2017-08-28 15:18:08,983 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:08,991 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:08,992 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node2', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:18:08,992 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:08,992 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:09,001 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 1, 'name': 'Node2'}, 'Node1') 2017-08-28 15:18:09,001 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:09,002 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node4') 2017-08-28 15:18:09,016 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:18:09,016 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node1') 2017-08-28 15:18:09,018 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:09,033 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-08-28 15:18:09,033 | DEBUG | primary_selector.py ( 265) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:09,033 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:09,038 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 3) 2017-08-28 15:18:09,039 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node2 2017-08-28 15:18:09,039 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node2 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:18:09,040 | DEBUG | primary_selector.py ( 280) | _startSelection | Node2 starting selection 2017-08-28 15:18:09,051 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node2:0 for instance 0 (view 1) 2017-08-28 15:18:09,051 | INFO | node.py ( 503) | start_participating | Node2 started participating 2017-08-28 15:18:09,055 | DEBUG | replica.py ( 406) | primaryName | Node2:0 setting primaryName for view no 1 to: Node2:0 2017-08-28 15:18:09,056 | DEBUG | replica.py (1678) | _gc | Node2:0 cleaning up till (0, 6) 2017-08-28 15:18:09,056 | DEBUG | replica.py (1693) | _gc | Node2:0 found 6 3-phase keys to clean 2017-08-28 15:18:09,064 | DEBUG | replica.py (1695) | _gc | Node2:0 found 6 request keys to clean 2017-08-28 15:18:09,064 | DEBUG | replica.py ( 311) | h | Node2:0 set watermarks as 0 300 2017-08-28 15:18:09,064 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,066 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node3:1 for instance 1 (view 1) 2017-08-28 15:18:09,075 | DEBUG | replica.py ( 406) | primaryName | Node2:1 setting primaryName for view no 1 to: Node3:1 2017-08-28 15:18:09,076 | DEBUG | replica.py (1678) | _gc | Node2:1 cleaning up till (0, 6) 2017-08-28 15:18:09,076 | DEBUG | replica.py (1693) | _gc | Node2:1 found 6 3-phase keys to clean 2017-08-28 15:18:09,076 | DEBUG | replica.py (1695) | _gc | Node2:1 found 6 request keys to clean 2017-08-28 15:18:09,077 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) from previous checkpoints 2017-08-28 15:18:09,088 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) from previous checkpoints 2017-08-28 15:18:09,088 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) from previous checkpoints 2017-08-28 15:18:09,094 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) from previous checkpoints 2017-08-28 15:18:09,094 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) from previous checkpoints 2017-08-28 15:18:09,094 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) from previous checkpoints 2017-08-28 15:18:09,095 | DEBUG | replica.py ( 311) | h | Node2:1 set watermarks as 0 300 2017-08-28 15:18:09,095 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (0, 6) 2017-08-28 15:18:09,095 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (1, 0) 2017-08-28 15:18:09,101 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,108 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:18:09,108 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node2', 'viewNo': 1, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} because it already decided primary which is Node2:0 2017-08-28 15:18:09,142 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.976 seconds 2017-08-28 15:18:12,741 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:12,753 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,754 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,757 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:12,772 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:15,146 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 37 2017-08-28 15:18:15,149 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:18:15,155 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:15,156 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 45 to run in 60 seconds 2017-08-28 15:18:15,156 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-08-28 15:18:15,156 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:18:15,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-08-28 15:18:15,601 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:15,605 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:15,606 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:15,608 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:18:15,659 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:18:15,674 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933495609306 2017-08-28 15:18:15,682 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,683 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,692 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,693 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,693 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933495609306 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,708 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:15,715 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since not finalised 2017-08-28 15:18:15,725 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:15,732 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:18:15,744 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:18:15,744 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:18:15,745 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:18:15,757 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:18:15,765 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:18:15,783 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:15,784 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,785 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,792 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:18:15,794 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,812 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,813 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,819 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:15,820 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,845 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,846 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,846 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:18:15,860 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,861 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,867 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,877 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) to 0 replicas 2017-08-28 15:18:15,878 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,891 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,892 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:18:15,901 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933495609306, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:15,902 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:15,910 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:18:15,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 43 2017-08-28 15:18:15,923 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:15,923 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:15,935 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:15,936 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:15,936 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:15,936 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:15,939 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-08-28 15:18:15,954 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:0 creating batch 1 for ledger 0 with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:15,959 | WARNING | replica.py ( 655) | processReqDuringBatch | Node2:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'reqId': 1503933495609306, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}, will reject 2017-08-28 15:18:15,970 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:0 created a PRE-PREPARE with 0 requests for ledger 0 2017-08-28 15:18:15,970 | DEBUG | replica.py (2133) | send | Node2:0 sending PrePrepare 2017-08-28 15:18:15,970 | TRACE | replica.py (2134) | send | Node2:0 sending PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933495, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:18:15,971 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933495, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:15,996 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933495609306,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REJECT","reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:16,009 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933495, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:16,010 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,022 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933495, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:18:16,035 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPREPARE', 'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'ppTime': 1503933495, 'discarded': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,036 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933495, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:18:16,049 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node4 2017-08-28 15:18:16,050 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node4 2017-08-28 15:18:16,055 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node1 2017-08-28 15:18:16,056 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node1 2017-08-28 15:18:16,057 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node3 2017-08-28 15:18:16,063 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933495609306]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933495,"discarded":0,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","ledgerId":0}' to Node3 2017-08-28 15:18:16,077 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:1 received PRE-PREPARE(1, 1) from Node3:1 at 249792.22596612 2017-08-28 15:18:16,078 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(1, 1) at 249792.226848948 2017-08-28 15:18:16,087 | DEBUG | replica.py (2133) | send | Node2:1 sending Prepare 2017-08-28 15:18:16,090 | TRACE | replica.py (2134) | send | Node2:1 sending PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,091 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,095 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933495, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} with state root None 2017-08-28 15:18:16,096 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(1, 1) 2017-08-28 15:18:16,102 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:16,120 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,132 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,133 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,134 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,149 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node4 2017-08-28 15:18:16,150 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node4 2017-08-28 15:18:16,165 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node1 2017-08-28 15:18:16,165 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node1 2017-08-28 15:18:16,167 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node3 2017-08-28 15:18:16,168 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933495,"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761"}' to Node3 2017-08-28 15:18:16,170 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 1) from Node1:1 2017-08-28 15:18:16,180 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(1, 1) at 249792.328801025 2017-08-28 15:18:16,180 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:18:16,180 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,187 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:18:16,192 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,193 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,204 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,206 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,212 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,230 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:16,240 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 1) from Node4:1 2017-08-28 15:18:16,240 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:18:16,240 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,258 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,259 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,259 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'op': 'PREPARE', 'ppTime': 1503933495, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,259 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,291 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,292 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,292 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node1 2017-08-28 15:18:16,294 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node1 2017-08-28 15:18:16,300 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,300 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,316 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 1) from Node3:0 2017-08-28 15:18:16,320 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,320 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,321 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:16,340 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,340 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,340 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,341 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,346 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,347 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,366 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 1) from Node4:0 2017-08-28 15:18:16,375 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 1) at 249792.523932126 2017-08-28 15:18:16,382 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:18:16,382 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,383 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:18:16,389 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,389 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 1) from Node1:0 2017-08-28 15:18:16,397 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:18:16,397 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,398 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:16,411 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,416 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,417 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,417 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,426 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,427 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,446 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:16,453 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 1) from Node4:1 2017-08-28 15:18:16,454 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:18:16,454 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,461 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 1) from Node1:1 2017-08-28 15:18:16,462 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:18:16,463 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (1, 1) 2017-08-28 15:18:16,463 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (1, 1) 2017-08-28 15:18:16,464 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:18:16,464 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933495, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'instId': 1, 'viewNo': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:18:16,464 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (1, 1) 2017-08-28 15:18:16,479 | DEBUG | replica.py (1540) | _newCheckpointState | Node2:1 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,480 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,481 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,481 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,493 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,493 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,509 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,509 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,517 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node1 2017-08-28 15:18:16,518 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node1 2017-08-28 15:18:16,524 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,525 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,526 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:18:16,539 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 1} because already ordered 3 phase message 2017-08-28 15:18:16,544 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:16,544 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,557 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,557 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,557 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,564 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,564 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,587 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node3:0 2017-08-28 15:18:16,591 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} commits where f is 1 2017-08-28 15:18:16,592 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,595 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node4:0 2017-08-28 15:18:16,595 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:18:16,596 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (1, 1) 2017-08-28 15:18:16,609 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (1, 1) 2017-08-28 15:18:16,611 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:18:16,617 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933495, 'reqIdr': [], 'instId': 0, 'viewNo': 1, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:18:16,617 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (1, 1) 2017-08-28 15:18:16,617 | DEBUG | replica.py (1540) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,618 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,619 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:16,636 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,636 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 1}, 'Node1') 2017-08-28 15:18:16,637 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} 2017-08-28 15:18:16,658 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 1 1 of 0 requests 2017-08-28 15:18:16,668 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:16,697 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:16,698 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 1} because already ordered 3 phase message 2017-08-28 15:18:16,735 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.097 seconds 2017-08-28 15:18:22,296 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:22,297 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:22,305 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:22,319 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:22,319 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:23,245 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:18:23,251 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933503177919 2017-08-28 15:18:23,251 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,263 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,270 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,285 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,286 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933503177919 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,295 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:23,296 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since not finalised 2017-08-28 15:18:23,297 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,311 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:18:23,324 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:18:23,329 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:18:23,338 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:18:23,341 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:18:23,345 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:18:23,360 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:23,361 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,375 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,382 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node4') 2017-08-28 15:18:23,383 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,400 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,401 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,412 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1') 2017-08-28 15:18:23,413 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,414 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,427 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,431 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3') 2017-08-28 15:18:23,432 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,444 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,451 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,452 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) to 0 replicas 2017-08-28 15:18:23,466 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,467 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,478 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:18:23,479 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933503177919, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} 2017-08-28 15:18:23,483 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:18:23,490 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:18:23,515 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:0 creating batch 2 for ledger 0 with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:23,516 | WARNING | replica.py ( 655) | processReqDuringBatch | Node2:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'reqId': 1503933503177919, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}, will reject 2017-08-28 15:18:23,523 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:0 created a PRE-PREPARE with 0 requests for ledger 0 2017-08-28 15:18:23,524 | DEBUG | replica.py (2133) | send | Node2:0 sending PrePrepare 2017-08-28 15:18:23,527 | TRACE | replica.py (2134) | send | Node2:0 sending PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933503, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 2} 2017-08-28 15:18:23,534 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933503, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 2} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:23,553 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933503177919,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REJECT","reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,569 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933503, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:23,570 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:23,580 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933503, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 2} 2017-08-28 15:18:23,580 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPREPARE', 'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'ppTime': 1503933503, 'discarded': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:23,580 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933503, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 2} 2017-08-28 15:18:23,603 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node4 2017-08-28 15:18:23,606 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node4 2017-08-28 15:18:23,607 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node1 2017-08-28 15:18:23,616 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node1 2017-08-28 15:18:23,620 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node3 2017-08-28 15:18:23,621 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933503177919]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":2,"ppTime":1503933503,"discarded":0,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","ledgerId":0}' to Node3 2017-08-28 15:18:23,622 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:1 received PRE-PREPARE(1, 2) from Node3:1 at 249799.770978174 2017-08-28 15:18:23,639 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(1, 2) at 249799.788401443 2017-08-28 15:18:23,640 | DEBUG | replica.py (2133) | send | Node2:1 sending Prepare 2017-08-28 15:18:23,640 | TRACE | replica.py (2134) | send | Node2:1 sending PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,640 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,651 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933503, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 2} with state root None 2017-08-28 15:18:23,652 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(1, 2) 2017-08-28 15:18:23,678 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:23,698 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node4 2017-08-28 15:18:23,698 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node4 2017-08-28 15:18:23,707 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node1 2017-08-28 15:18:23,715 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node1 2017-08-28 15:18:23,716 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node3 2017-08-28 15:18:23,716 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":2,"ppTime":1503933503,"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5"}' to Node3 2017-08-28 15:18:23,743 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.509 seconds 2017-08-28 15:18:23,764 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:23,764 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,768 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:23,768 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,768 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,776 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:23,776 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,779 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 2) from Node4:1 2017-08-28 15:18:23,806 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(1, 2) at 249799.955372626 2017-08-28 15:18:23,807 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:18:23,807 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:23,817 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:18:23,825 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,825 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 2) from Node1:1 2017-08-28 15:18:23,826 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': None, 'ppSeqNo': 2} 2017-08-28 15:18:23,826 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,841 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:23,853 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:23,874 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'op': 'PREPARE', 'ppTime': 1503933503, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:23,874 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:23,886 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:23,897 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 2) from Node3:0 2017-08-28 15:18:23,898 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:23,898 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,909 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:23,910 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:23,920 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:23,920 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:23,932 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:23,933 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:23,950 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node1 2017-08-28 15:18:23,951 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node1 2017-08-28 15:18:23,951 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:23,955 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:23,967 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 2) from Node1:0 2017-08-28 15:18:23,971 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 2) at 249800.119749012 2017-08-28 15:18:23,971 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:18:23,974 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:23,975 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:18:23,975 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,992 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:18:23,993 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:23,993 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:24,004 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,004 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:24,010 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,010 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:24,032 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:18:24,040 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 2) from Node4:0 2017-08-28 15:18:24,040 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 2} 2017-08-28 15:18:24,051 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:24,052 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 2) from Node1:1 2017-08-28 15:18:24,052 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:18:24,056 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,057 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:24,067 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,068 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:24,068 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,092 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:24,093 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:24,098 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node1 2017-08-28 15:18:24,098 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node1 2017-08-28 15:18:24,099 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:24,107 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:24,108 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 2) from Node3:1 2017-08-28 15:18:24,119 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:18:24,123 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (1, 2) 2017-08-28 15:18:24,123 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (1, 2) 2017-08-28 15:18:24,123 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:18:24,124 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933503, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'instId': 1, 'viewNo': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 2} 2017-08-28 15:18:24,124 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (1, 2) 2017-08-28 15:18:24,125 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,143 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:18:24,144 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,144 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,152 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,152 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,153 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:24,162 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,163 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,167 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node1') 2017-08-28 15:18:24,171 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,178 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:18:24,193 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 2) from Node3:0 2017-08-28 15:18:24,193 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} commits where f is 1 2017-08-28 15:18:24,193 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,203 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 2) from Node1:0 2017-08-28 15:18:24,206 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:18:24,206 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (1, 2) 2017-08-28 15:18:24,207 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (1, 2) 2017-08-28 15:18:24,207 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:18:24,215 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933503, 'reqIdr': [], 'instId': 0, 'viewNo': 1, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 2} 2017-08-28 15:18:24,216 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (1, 2) 2017-08-28 15:18:24,216 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,222 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 2} because already ordered 3 phase message 2017-08-28 15:18:24,223 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:24,232 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,232 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,244 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} 2017-08-28 15:18:24,260 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 1 2 of 0 requests 2017-08-28 15:18:24,265 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:24,266 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:24,266 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 2} because already ordered 3 phase message 2017-08-28 15:18:24,308 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.555 seconds 2017-08-28 15:18:25,957 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 46 2017-08-28 15:18:25,959 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:25,964 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:25,965 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:25,965 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:25,966 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:25,966 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:25,966 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-08-28 15:18:28,114 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:28,114 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,115 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,122 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:28,132 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:30,604 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:30,624 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:30,625 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:30,625 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:18:35,967 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 47 2017-08-28 15:18:35,970 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:35,971 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:35,971 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:35,976 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:35,990 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:35,996 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:35,997 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-08-28 15:18:37,689 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:37,700 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,700 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,715 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:37,719 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:42,796 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:42,801 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,801 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,816 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:42,826 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:45,629 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:45,629 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:18:45,631 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:45,647 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:18:45,998 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 48 2017-08-28 15:18:45,998 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:45,998 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:46,003 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:46,007 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:46,008 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:46,021 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:46,021 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-08-28 15:18:52,346 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:52,346 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,347 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,368 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:52,373 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:56,046 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 49 2017-08-28 15:18:56,050 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:18:56,051 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:56,061 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:18:56,061 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:56,062 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:56,066 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:18:56,070 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-08-28 15:18:58,130 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:18:58,131 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,144 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,146 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:58,158 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:00,638 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:00,655 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:00,659 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:00,670 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:19:06,077 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 50 2017-08-28 15:19:06,077 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:06,078 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:06,078 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:06,078 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:06,079 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:06,079 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:06,079 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 51 to run in 10 seconds 2017-08-28 15:19:07,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action _check_view_change_completed with id 44 2017-08-28 15:19:07,241 | DEBUG | node.py ( 993) | _check_view_change_completed | Node2 running the scheduled check for view change completion 2017-08-28 15:19:07,241 | DEBUG | node.py ( 995) | _check_view_change_completed | Node2 already completion view change 2017-08-28 15:19:07,644 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:07,645 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,646 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,649 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:07,667 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:12,799 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:12,803 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,804 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,821 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:12,822 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:15,166 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 45 2017-08-28 15:19:15,167 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:19:15,171 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:15,171 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 52 to run in 60 seconds 2017-08-28 15:19:15,174 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-08-28 15:19:15,174 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:19:15,184 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-08-28 15:19:15,670 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:15,681 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:15,681 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:15,683 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:19:16,082 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 51 2017-08-28 15:19:16,083 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:16,086 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:16,086 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:16,086 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:16,088 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:16,105 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:16,106 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-08-28 15:19:22,360 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:22,363 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,369 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,377 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:22,387 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:26,112 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 53 2017-08-28 15:19:26,112 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:26,115 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:26,115 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:26,116 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:26,123 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:26,131 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:26,132 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-08-28 15:19:28,055 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:28,068 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,068 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,073 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:28,083 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:30,676 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:30,679 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:30,682 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:30,691 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:19:36,154 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 54 2017-08-28 15:19:36,154 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:36,157 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:36,159 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:36,164 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:36,171 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:36,172 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:36,183 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-08-28 15:19:37,712 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:37,713 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,721 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,736 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:37,736 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:42,802 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:42,814 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,817 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,833 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:42,837 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:45,712 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:45,718 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:19:45,720 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:45,721 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:19:46,219 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 55 2017-08-28 15:19:46,224 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:46,224 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:46,224 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:46,230 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:46,230 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:46,231 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:46,231 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-08-28 15:19:52,370 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:52,374 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,374 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,389 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:52,396 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:56,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 56 2017-08-28 15:19:56,233 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:19:56,238 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:56,239 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:19:56,241 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:56,241 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:56,241 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:19:56,242 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-08-28 15:19:58,113 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:19:58,114 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,125 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,138 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:58,139 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:00,729 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:00,735 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:00,735 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:00,740 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:20:06,243 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 57 2017-08-28 15:20:06,258 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:06,258 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:06,259 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:06,260 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:06,260 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:06,263 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:06,279 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-08-28 15:20:07,681 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:07,683 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,684 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,708 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:07,708 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:12,818 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:12,822 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,823 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,834 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:12,855 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:15,181 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 52 2017-08-28 15:20:15,185 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:20:15,186 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:15,187 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds 2017-08-28 15:20:15,191 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-08-28 15:20:15,192 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:20:15,192 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-08-28 15:20:15,734 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:15,737 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:15,737 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:15,738 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:20:16,290 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 58 2017-08-28 15:20:16,304 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:16,307 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:16,308 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:16,308 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:16,309 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:16,309 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:16,315 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-08-28 15:20:22,410 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:22,412 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,420 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,438 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:22,439 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:26,333 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 60 2017-08-28 15:20:26,336 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:26,339 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:26,340 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:26,342 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:26,344 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:26,347 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:26,356 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 61 to run in 10 seconds 2017-08-28 15:20:28,088 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:28,100 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,100 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,110 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:28,112 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:30,759 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:30,766 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:30,768 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:30,769 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:20:36,367 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 61 2017-08-28 15:20:36,367 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:36,368 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:36,369 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:36,369 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:36,370 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:36,370 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:36,370 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-08-28 15:20:37,685 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:37,703 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,704 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,706 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:37,719 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:42,855 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:42,868 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,868 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,877 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:42,885 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:45,770 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:45,787 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:20:45,788 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:45,788 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:20:46,393 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 62 2017-08-28 15:20:46,404 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:46,404 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:46,406 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:46,407 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:46,411 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:46,416 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:46,416 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-08-28 15:20:52,417 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:52,433 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,433 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,456 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:52,457 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:56,430 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 63 2017-08-28 15:20:56,442 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:20:56,442 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:56,442 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:20:56,442 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:56,442 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:56,443 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:20:56,443 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-08-28 15:20:58,091 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:20:58,108 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,108 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,110 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:58,123 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:00,804 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:00,804 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:00,804 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:00,812 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:21:06,449 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 64 2017-08-28 15:21:06,452 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:06,452 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:06,452 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:06,455 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:06,456 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:06,456 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:06,456 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-08-28 15:21:07,793 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:07,794 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,795 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,805 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:07,817 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:12,875 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:12,888 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,888 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,910 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:12,910 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:15,207 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 59 2017-08-28 15:21:15,208 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:21:15,208 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:15,209 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 66 to run in 60 seconds 2017-08-28 15:21:15,212 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 9 2017-08-28 15:21:15,212 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:21:15,213 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-08-28 15:21:15,818 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:15,828 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:15,833 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:15,833 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:21:16,470 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 65 2017-08-28 15:21:16,482 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:16,483 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:16,484 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:16,484 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:16,488 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:16,488 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:16,488 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-08-28 15:21:20,538 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.139 seconds 2017-08-28 15:21:22,450 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:22,465 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,466 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,478 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:22,481 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:26,493 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 67 2017-08-28 15:21:26,495 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:26,496 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:26,511 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:26,511 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:26,516 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:26,517 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:26,517 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-08-28 15:21:28,148 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:28,149 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,150 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,173 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:28,173 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:30,825 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:30,826 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:30,839 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:30,844 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:21:36,519 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 68 2017-08-28 15:21:36,523 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:36,538 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:36,539 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:36,539 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:36,542 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:36,542 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:36,542 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-08-28 15:21:37,747 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:37,760 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,760 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,769 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:37,783 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:42,897 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:42,903 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,915 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,917 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:42,935 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:45,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:45,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:21:45,840 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:45,841 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:21:46,557 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 69 2017-08-28 15:21:46,557 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:46,574 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:46,574 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:46,574 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:46,574 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:46,577 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:46,591 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-08-28 15:21:52,517 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:52,527 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,532 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,544 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:52,544 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:56,624 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 70 2017-08-28 15:21:56,626 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:21:56,626 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:56,626 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:21:56,628 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:56,628 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:56,629 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:21:56,629 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-08-28 15:21:58,285 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:21:58,290 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,294 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,297 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:58,320 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:00,855 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:00,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:00,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:00,856 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:22:06,648 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 71 2017-08-28 15:22:06,654 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:06,655 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:06,661 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:06,662 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:06,666 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:06,666 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:06,671 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-08-28 15:22:07,931 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:07,932 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,932 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,934 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:07,934 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:12,924 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:12,925 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,925 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,944 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:12,951 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:15,225 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 66 2017-08-28 15:22:15,235 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:22:15,236 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:15,239 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 73 to run in 60 seconds 2017-08-28 15:22:15,241 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-08-28 15:22:15,244 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:22:15,257 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-08-28 15:22:15,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:15,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:15,883 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:15,886 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:22:16,720 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 72 2017-08-28 15:22:16,721 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:16,721 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:16,722 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:16,722 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:16,723 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:16,723 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:16,724 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-08-28 15:22:22,577 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:22,578 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,598 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,641 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:22,641 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:26,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 74 2017-08-28 15:22:26,768 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:26,769 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:26,771 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:26,781 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:26,781 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:26,784 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:26,789 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-08-28 15:22:28,124 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:28,128 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,131 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,150 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:28,151 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:30,911 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:30,925 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:30,925 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:30,926 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:22:36,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 75 2017-08-28 15:22:36,812 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:36,812 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:36,812 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:36,814 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:36,829 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:36,830 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:36,830 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-08-28 15:22:37,966 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:37,972 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,972 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,992 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:37,992 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:42,967 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:42,968 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,968 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,976 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:42,976 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:45,934 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:45,935 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:22:45,939 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:45,940 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:22:46,848 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 76 2017-08-28 15:22:46,848 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:46,848 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:46,854 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:46,855 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:46,855 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:46,856 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:46,856 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 77 to run in 10 seconds 2017-08-28 15:22:52,572 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:52,573 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,583 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,604 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:52,610 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:56,880 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 77 2017-08-28 15:22:56,887 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:22:56,888 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:56,888 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:22:56,888 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:56,898 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:56,898 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:22:56,899 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-08-28 15:22:58,125 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:22:58,128 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,136 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,138 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:58,156 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:00,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:00,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:00,963 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:00,964 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:23:06,923 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 78 2017-08-28 15:23:06,932 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:06,933 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:06,933 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:06,933 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:06,942 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:06,942 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:06,947 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-08-28 15:23:07,880 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:07,884 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,900 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,902 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:07,919 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:12,991 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:12,994 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:12,995 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:13,008 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:13,023 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:15,266 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 73 2017-08-28 15:23:15,270 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:23:15,270 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:15,270 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 80 to run in 60 seconds 2017-08-28 15:23:15,270 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 11 2017-08-28 15:23:15,270 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:23:15,270 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-08-28 15:23:15,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:15,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:15,995 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:15,995 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:23:16,954 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 79 2017-08-28 15:23:16,964 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:16,968 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:16,968 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:16,968 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:16,969 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:16,972 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:16,990 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-08-28 15:23:22,601 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:22,612 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,612 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,634 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:22,635 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:27,010 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 81 2017-08-28 15:23:27,011 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:27,011 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:27,014 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:27,014 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:27,014 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:27,015 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:27,015 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-08-28 15:23:28,290 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:28,292 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,304 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,315 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:28,316 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:31,006 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:31,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:31,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:31,034 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:23:37,017 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 82 2017-08-28 15:23:37,034 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:37,043 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:37,043 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:37,044 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:37,044 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:37,044 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:37,045 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-08-28 15:23:37,877 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:37,880 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,885 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,892 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:37,897 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:42,979 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:42,988 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:42,995 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:43,002 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:43,011 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:46,018 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:46,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:23:46,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:46,028 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:23:47,066 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 83 2017-08-28 15:23:47,078 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:47,078 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:47,079 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:47,081 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:47,086 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:47,086 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:47,087 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-08-28 15:23:52,585 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:52,589 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,589 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,590 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:52,591 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:57,106 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 84 2017-08-28 15:23:57,107 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:23:57,107 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:57,120 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:23:57,120 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:57,120 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:57,120 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:23:57,121 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 85 to run in 10 seconds 2017-08-28 15:23:58,152 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:23:58,159 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,165 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,173 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:58,178 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:01,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:01,031 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:01,031 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:01,032 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:24:07,125 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 85 2017-08-28 15:24:07,128 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:07,129 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:07,135 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:07,136 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:07,136 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:07,136 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:07,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-08-28 15:24:07,819 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:07,822 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,825 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,845 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:07,846 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:12,984 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:12,990 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:12,990 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:13,002 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:13,003 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:15,287 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 80 2017-08-28 15:24:15,299 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:24:15,300 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:15,300 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 87 to run in 60 seconds 2017-08-28 15:24:15,300 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 12 2017-08-28 15:24:15,311 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:24:15,315 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-08-28 15:24:16,040 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:16,054 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:16,055 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:16,060 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:24:17,152 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 86 2017-08-28 15:24:17,152 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:17,153 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:17,153 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:17,159 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:17,159 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:17,165 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:17,166 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-08-28 15:24:22,615 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:22,620 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,624 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,629 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:22,643 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:27,181 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 88 2017-08-28 15:24:27,200 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:27,201 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:27,201 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:27,201 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:27,202 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:27,202 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:27,204 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-08-28 15:24:28,160 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:28,161 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,161 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,176 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:28,187 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:31,050 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:31,059 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:31,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:31,061 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:24:37,216 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 89 2017-08-28 15:24:37,217 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:37,217 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:37,217 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:37,218 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:37,218 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:37,218 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:37,219 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-08-28 15:24:37,913 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:37,914 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,915 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,935 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:37,935 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:43,008 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:43,009 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,009 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,033 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:43,033 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:46,059 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:46,063 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:24:46,065 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:46,069 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:24:47,229 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 90 2017-08-28 15:24:47,230 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:47,230 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:47,231 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:47,232 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:47,233 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:47,235 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:47,235 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-08-28 15:24:52,614 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:52,627 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,628 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,629 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:52,645 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:57,249 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 91 2017-08-28 15:24:57,249 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:24:57,250 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:57,251 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:24:57,255 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:57,256 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:57,263 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:24:57,270 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-08-28 15:24:58,244 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:24:58,244 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,245 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,261 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:58,271 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:01,076 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:01,078 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:01,088 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:01,095 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:25:07,295 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 92 2017-08-28 15:25:07,296 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:07,296 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:07,301 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:07,301 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:07,302 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:07,302 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:07,304 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 93 to run in 10 seconds 2017-08-28 15:25:08,008 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:08,030 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:08,036 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:08,045 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:08,054 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:13,049 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:13,064 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,064 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,080 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:13,086 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:15,338 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 87 2017-08-28 15:25:15,349 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node2 checking its request amount 2017-08-28 15:25:15,350 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:15,350 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 94 to run in 60 seconds 2017-08-28 15:25:15,350 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 13 2017-08-28 15:25:15,350 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:25:15,351 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-08-28 15:25:16,099 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:16,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:16,106 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:16,106 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:25:17,336 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 93 2017-08-28 15:25:17,337 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:17,339 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:17,344 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:17,344 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:17,344 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:17,346 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:17,346 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-08-28 15:25:22,632 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:22,642 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,642 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,644 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:22,662 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:27,370 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 95 2017-08-28 15:25:27,375 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:27,376 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:27,376 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:27,381 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:27,382 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:27,382 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:27,382 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-08-28 15:25:28,336 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:28,348 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,349 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,350 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:28,367 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:31,108 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:31,126 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:31,127 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:31,131 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:25:37,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 96 2017-08-28 15:25:37,386 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:37,395 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:37,396 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:37,396 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:37,402 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:37,403 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:37,405 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-08-28 15:25:37,913 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:37,914 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,915 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,933 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:37,933 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:43,080 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:43,080 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,081 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,096 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:43,096 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:46,113 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:46,118 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-08-28 15:25:46,119 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:46,128 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-08-28 15:25:47,406 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 97 2017-08-28 15:25:47,406 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:47,423 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:47,423 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:47,425 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:47,425 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node2 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:47,425 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:47,425 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-08-28 15:25:47,464 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:25:47,473 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on request 1503933947440487 2017-08-28 15:25:47,489 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,490 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,508 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,509 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,509 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating 6XiqM9DwYDgFnmyQQBFr72 request 1503933947440487 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,519 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:47,522 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since not finalised 2017-08-28 15:25:47,522 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,540 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:25:47,540 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:25:47,552 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:25:47,552 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:25:47,561 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:25:47,572 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:25:47,574 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:47,597 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,598 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,603 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}}, 'Node4') 2017-08-28 15:25:47,607 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,615 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,626 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,627 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}}, 'Node1') 2017-08-28 15:25:47,634 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,651 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,652 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,653 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}}, 'Node3') 2017-08-28 15:25:47,653 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,660 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,671 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,675 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) to 0 replicas 2017-08-28 15:25:47,675 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,686 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,693 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:25:47,694 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933947440487, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72'}} 2017-08-28 15:25:47,704 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:47,704 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:25:47,718 | DEBUG | replica.py ( 665) | create3PCBatch | Node2:0 creating batch 3 for ledger 0 with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:47,733 | WARNING | replica.py ( 655) | processReqDuringBatch | Node2:0 encountered exception UnauthorizedClientRequest('6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node',) while processing Request: {'reqId': 1503933947440487, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}, will reject 2017-08-28 15:25:47,734 | DEBUG | replica.py ( 698) | create3PCBatch | Node2:0 created a PRE-PREPARE with 0 requests for ledger 0 2017-08-28 15:25:47,749 | DEBUG | replica.py (2133) | send | Node2:0 sending PrePrepare 2017-08-28 15:25:47,750 | TRACE | replica.py (2134) | send | Node2:0 sending PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933947, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 3} 2017-08-28 15:25:47,750 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933947, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 3} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:47,763 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933947440487,"identifier":"6XiqM9DwYDgFnmyQQBFr72","op":"REJECT","reason":"client request invalid: UnauthorizedClientRequest(\'6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node\',)"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,783 | DEBUG | node.py (2566) | send | Node2 sending message PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933947, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:47,807 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node4 2017-08-28 15:25:47,811 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node4 2017-08-28 15:25:47,814 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node1 2017-08-28 15:25:47,814 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node1 2017-08-28 15:25:47,816 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node3 2017-08-28 15:25:47,826 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPREPARE","viewNo":1,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"reqIdr":[["6XiqM9DwYDgFnmyQQBFr72",1503933947440487]],"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":3,"ppTime":1503933947,"discarded":0,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","ledgerId":0}' to Node3 2017-08-28 15:25:47,844 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:47,845 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933947, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 3} 2017-08-28 15:25:47,854 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPREPARE', 'ledgerId': 0, 'instId': 1, 'txnRootHash': None, 'ppTime': 1503933947, 'discarded': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:47,855 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933947, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 3} 2017-08-28 15:25:47,865 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:1 received PRE-PREPARE(1, 3) from Node3:1 at 250244.014448508 2017-08-28 15:25:47,878 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(1, 3) at 250244.026919889 2017-08-28 15:25:47,878 | DEBUG | replica.py (2133) | send | Node2:1 sending Prepare 2017-08-28 15:25:47,887 | TRACE | replica.py (2134) | send | Node2:1 sending PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:47,887 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:47,888 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933947, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 3} with state root None 2017-08-28 15:25:47,888 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(1, 3) 2017-08-28 15:25:47,916 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:47,943 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node4 2017-08-28 15:25:47,944 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node4 2017-08-28 15:25:47,944 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node1 2017-08-28 15:25:47,953 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node1 2017-08-28 15:25:47,953 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node3 2017-08-28 15:25:47,953 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":1,"txnRootHash":null,"ppSeqNo":3,"ppTime":1503933947,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}' to Node3 2017-08-28 15:25:47,969 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:47,970 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:47,982 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'op': 'PREPARE', 'ppTime': 1503933947, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:47,982 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,007 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 3) from Node3:0 2017-08-28 15:25:48,008 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,008 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,016 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:48,017 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}']} 2017-08-28 15:25:48,028 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}'], 'signature': None}, 'Node1') 2017-08-28 15:25:48,034 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', '{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}']} 2017-08-28 15:25:48,035 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,036 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,038 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,038 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:48,045 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'viewNo': 1, 'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,046 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:48,059 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}']} 2017-08-28 15:25:48,062 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}']}, 'Node4') 2017-08-28 15:25:48,063 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}']} 2017-08-28 15:25:48,064 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,075 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,078 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,091 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:48,092 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PREPARE', 'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,092 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:48,114 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 3) from Node1:0 2017-08-28 15:25:48,124 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 3) at 250244.272729262 2017-08-28 15:25:48,124 | DEBUG | replica.py (2133) | send | Node2:0 sending Commit 2017-08-28 15:25:48,124 | TRACE | replica.py (2134) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,130 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:25:48,136 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,136 | DEBUG | replica.py ( 865) | processPrepare | Node2:0 received PREPARE(1, 3) from Node4:0 2017-08-28 15:25:48,136 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 3} 2017-08-28 15:25:48,147 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,147 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 3) from Node1:1 2017-08-28 15:25:48,147 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(1, 3) at 250244.296587172 2017-08-28 15:25:48,148 | DEBUG | replica.py (2133) | send | Node2:1 sending Commit 2017-08-28 15:25:48,148 | TRACE | replica.py (2134) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,158 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:25:48,158 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,165 | DEBUG | replica.py ( 865) | processPrepare | Node2:1 received PREPARE(1, 3) from Node4:1 2017-08-28 15:25:48,165 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'instId': 1, 'viewNo': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'txnRootHash': None, 'ppSeqNo': 3} 2017-08-28 15:25:48,166 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,192 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:48,193 | DEBUG | node.py (2566) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:48,214 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:48,216 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}']) 2017-08-28 15:25:48,224 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' 2017-08-28 15:25:48,231 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' to Node4 2017-08-28 15:25:48,232 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:48,232 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}']) 2017-08-28 15:25:48,232 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' 2017-08-28 15:25:48,242 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' to Node1 2017-08-28 15:25:48,243 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:48,243 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}']) 2017-08-28 15:25:48,251 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' 2017-08-28 15:25:48,253 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"instId\\":0,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}","{\\"instId\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1,\\"ppSeqNo\\":3}"],"signature":null}' to Node3 2017-08-28 15:25:48,269 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:48,270 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,282 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,283 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,291 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}']} 2017-08-28 15:25:48,292 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}']}, 'Node4') 2017-08-28 15:25:48,292 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}']} 2017-08-28 15:25:48,292 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,293 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,301 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,310 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,310 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,317 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,317 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:25:48,318 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']}, 'Node3') 2017-08-28 15:25:48,318 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:25:48,319 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,330 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:48,330 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,330 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,337 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:48,338 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,351 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 3) from Node4:0 2017-08-28 15:25:48,371 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:25:48,371 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,377 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 3) from Node3:0 2017-08-28 15:25:48,377 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-08-28 15:25:48,377 | DEBUG | replica.py (1437) | doOrder | Node2:0 ordering COMMIT (1, 3) 2017-08-28 15:25:48,378 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:0 set last ordered as (1, 3) 2017-08-28 15:25:48,380 | DEBUG | replica.py (2133) | send | Node2:0 sending Ordered 2017-08-28 15:25:48,387 | TRACE | replica.py (2134) | send | Node2:0 sending ORDERED{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'ppTime': 1503933947, 'reqIdr': [], 'instId': 0, 'viewNo': 1, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 3} 2017-08-28 15:25:48,387 | DEBUG | replica.py (1469) | order_3pc_key | Node2:0 ordered request (1, 3) 2017-08-28 15:25:48,387 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,399 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 3) from Node1:1 2017-08-28 15:25:48,399 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} commits where f is 1 2017-08-28 15:25:48,402 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,404 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 3) from Node4:1 2017-08-28 15:25:48,404 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-08-28 15:25:48,410 | DEBUG | replica.py (1437) | doOrder | Node2:1 ordering COMMIT (1, 3) 2017-08-28 15:25:48,411 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (1, 3) 2017-08-28 15:25:48,411 | DEBUG | replica.py (2133) | send | Node2:1 sending Ordered 2017-08-28 15:25:48,416 | TRACE | replica.py (2134) | send | Node2:1 sending ORDERED{'stateRootHash': None, 'ppTime': 1503933947, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'instId': 1, 'viewNo': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 3} 2017-08-28 15:25:48,417 | DEBUG | replica.py (1469) | order_3pc_key | Node2:1 ordered request (1, 3) 2017-08-28 15:25:48,417 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,420 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 1, 'ppSeqNo': 3} because already ordered 3 phase message 2017-08-28 15:25:48,421 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:48,433 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,433 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 1, 'ppSeqNo': 3}, 'Node1') 2017-08-28 15:25:48,446 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} 2017-08-28 15:25:48,454 | DEBUG | node.py (1828) | processOrdered | Node2 executing Ordered batch 1 3 of 0 requests 2017-08-28 15:25:48,473 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:25:48,474 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:48,474 | TRACE | node.py (1839) | processOrdered | Node2 got ordered requests from backup replica 1 2017-08-28 15:25:48,475 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 1, 'ppSeqNo': 3} because already ordered 3 phase message 2017-08-28 15:25:48,526 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.122 seconds 2017-08-28 15:25:52,674 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:52,681 | DEBUG | zstack.py ( 661) | handlePingPong | Node2 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,681 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,698 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node2 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:52,698 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:53,669 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:53,670 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,679 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 2, 'reason': 25}, 'Node4') 2017-08-28 15:25:53,680 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,680 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} from Node4 2017-08-28 15:25:53,687 | DEBUG | node.py (1939) | do_view_change_if_possible | Node2 has no quorum for view 2 2017-08-28 15:25:53,688 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node2 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:25:53,688 | INFO | node.py (1920) | processInstanceChange | VIEW CHANGE: Node2 found master degraded after receiving instance change message from Node4 2017-08-28 15:25:53,688 | DEBUG | throttler.py ( 31) | acquire | now: 250249.837487742, len(actionsLog): 0 2017-08-28 15:25:53,693 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:25:53,693 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:25:53,693 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 2 since Primary of master protocol instance degraded the performance 2017-08-28 15:25:53,694 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249431.217309321, 249431.221819749] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 2.1880997460102662} master request latencies: {} client avg request latencies: [{}, {'V4SGRU86Z58d6TV7PBUe6f': (2, 0.6942614569998113), '6XiqM9DwYDgFnmyQQBFr72': (1, 0.7995768320106436)}] throughput: {0: 0, 1: 1.371052670459898} master throughput: 0 total requests: 6 avg backup throughput: 1.371052670459898 master throughput ratio: 0.0 2017-08-28 15:25:53,715 | DEBUG | node.py (2566) | send | Node2 sending message INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:53,719 | DEBUG | node.py (1939) | do_view_change_if_possible | Node2 has no quorum for view 2 2017-08-28 15:25:53,731 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node4 2017-08-28 15:25:53,732 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node4 2017-08-28 15:25:53,737 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node1 2017-08-28 15:25:53,737 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node1 2017-08-28 15:25:53,738 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node3 2017-08-28 15:25:53,744 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"INSTANCE_CHANGE","viewNo":2,"reason":25}' to Node3 2017-08-28 15:25:53,754 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:53,767 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,768 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 2, 'reason': 25}, 'Node3') 2017-08-28 15:25:53,774 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,774 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,775 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 2, 'reason': 25}, 'Node1') 2017-08-28 15:25:53,779 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} 2017-08-28 15:25:53,780 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} from Node3 2017-08-28 15:25:53,780 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 2 from 1 2017-08-28 15:25:53,781 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action _check_view_change_completed with id 99 to run in 60 seconds 2017-08-28 15:25:53,796 | DEBUG | replica.py ( 453) | on_view_change_start | Node2:0 setting last prepared for master to (1, 3) 2017-08-28 15:25:53,796 | DEBUG | node.py (2127) | startViewChange | Node2 resetting monitor stats after view change 2017-08-28 15:25:53,797 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:25:53,799 | DEBUG | replica.py ( 406) | primaryName | Node2:0 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,799 | DEBUG | replica.py ( 406) | primaryName | Node2:1 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,800 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node2 changed to view 2, will start catchup now 2017-08-28 15:25:53,808 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,808 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:53,809 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,809 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:53,809 | DEBUG | node.py (2173) | start_catchup | Node2 reverted 0 batches before starting catch up 2017-08-28 15:25:53,816 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:25:53,816 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 0 2017-08-28 15:25:53,823 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:53,824 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 0 2017-08-28 15:25:53,824 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:53,824 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 0 2017-08-28 15:25:53,825 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:53,826 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 0 2017-08-28 15:25:53,826 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:53,837 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:53,843 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:25:53,843 | DEBUG | node.py (1927) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:25:53,850 | DEBUG | node.py (1896) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} from Node1 2017-08-28 15:25:53,853 | INFO | message_processor.py ( 29) | discard | Node2 discarding message INSTANCE_CHANGE{'viewNo': 2, 'reason': 25} because Received instance change request with view no 2 which is not more than its view no 2 2017-08-28 15:25:53,855 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:53,868 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,868 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:53,868 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node4 2017-08-28 15:25:53,879 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:53,881 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,881 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:53,881 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-08-28 15:25:53,882 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:53,882 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,891 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:53,891 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-08-28 15:25:53,901 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:53,911 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:53,917 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:25:53,917 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:53,923 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,924 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:53,924 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,924 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,936 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:53,936 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,940 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:53,944 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:53,944 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:53,945 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,953 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:25:53,954 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,956 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,964 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:25:53,965 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,973 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:53,974 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:25:53,974 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:53,978 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,978 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:53,979 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,979 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,989 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:53,989 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,999 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,005 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,006 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,021 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,022 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,031 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,036 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,045 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,045 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:54,047 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node4 2017-08-28 15:25:54,060 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,060 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,061 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:54,071 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-08-28 15:25:54,072 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,080 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,081 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-08-28 15:25:54,091 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-08-28 15:25:54,107 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:54,116 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:54,124 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,125 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:54,125 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,127 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:25:54,131 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,135 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,143 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-08-28 15:25:54,144 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,144 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:25:54,153 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']}, 'Node3') 2017-08-28 15:25:54,153 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:25:54,162 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,163 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:54,168 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,169 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,169 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:54,169 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,171 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,180 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:25:54,183 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,191 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,192 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:54,192 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,192 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,195 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:54,204 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,205 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node1 2017-08-28 15:25:54,206 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,206 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,214 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node1 2017-08-28 15:25:54,214 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,215 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,226 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node3 2017-08-28 15:25:54,231 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,232 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,232 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:25:54,239 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:25:54,239 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,240 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,240 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,247 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,248 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,248 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:25:54,248 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:25:54,254 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:25:54,254 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,255 | DEBUG | node.py ( 923) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node4', 'Node1', 'Node3'} 2017-08-28 15:25:54,257 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,258 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-08-28 15:25:54,266 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:54,267 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-08-28 15:25:54,267 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,273 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-08-28 15:25:54,273 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,283 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-08-28 15:25:54,284 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:25:54,284 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:25:54,285 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node3 2017-08-28 15:25:54,296 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,297 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,297 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node4 2017-08-28 15:25:54,308 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,309 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,309 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:25:54,310 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None} from Node4 2017-08-28 15:25:54,321 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,321 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,321 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:25:54,339 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,340 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,340 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,351 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-08-28 15:25:54,352 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,352 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,358 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,366 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:25:54,367 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,367 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,367 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,368 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:25:54,388 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:54,389 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,400 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,400 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,406 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,411 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:25:54,415 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,418 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,419 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:25:54,419 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,419 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:54,425 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:25:54,426 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:54,426 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,428 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,438 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,440 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,440 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,440 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,448 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,449 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,465 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,471 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,489 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,489 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,498 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,508 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-08-28 15:25:54,509 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,511 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,521 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,526 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-08-28 15:25:54,552 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:54,553 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:54,559 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:25:54,560 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:54,560 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,563 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,569 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,570 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,579 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,580 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,580 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,592 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:54,593 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,594 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,595 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:25:54,613 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,613 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,614 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-08-28 15:25:54,614 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,620 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,629 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,636 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:25:54,637 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,644 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,645 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node1 2017-08-28 15:25:54,659 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,660 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,662 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,680 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,687 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-08-28 15:25:54,696 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node4 2017-08-28 15:25:54,698 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:54,714 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,728 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:25:54,729 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,740 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,747 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,747 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,751 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,752 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,759 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,760 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:25:54,770 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-08-28 15:25:54,770 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:25:54,776 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,777 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,777 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,777 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,787 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,787 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,788 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node4 2017-08-28 15:25:54,797 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,797 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,797 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node1'} that its ledger of type 2 is latest 2017-08-28 15:25:54,807 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} that it does not need catchup 2017-08-28 15:25:54,808 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,808 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,818 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,819 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,823 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,824 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:25:54,824 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:25:54,826 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:25:54,835 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,836 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:25:54,836 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-08-28 15:25:54,839 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:25:54,847 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-08-28 15:25:54,848 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:54,848 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-08-28 15:25:54,857 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,858 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-08-28 15:25:54,858 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,864 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-08-28 15:25:54,865 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:25:54,865 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-08-28 15:25:54,879 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:25:54,881 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node4 2017-08-28 15:25:54,881 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,891 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,892 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:25:54,896 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,905 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,905 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:25:54,906 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None} from Node3 2017-08-28 15:25:54,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,916 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,928 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:25:54,944 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,949 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,953 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:54,953 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:25:54,954 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:54,965 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,965 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:54,974 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:25:54,975 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,981 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,981 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:54,989 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:25:55,005 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:55,012 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,012 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:55,024 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,025 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,037 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:25:55,037 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,048 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,049 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:25:55,057 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,057 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:55,057 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']}, 'Node4') 2017-08-28 15:25:55,058 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:55,065 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,065 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,065 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,074 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,082 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,082 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,091 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:55,092 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-08-28 15:25:55,092 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:55,092 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,100 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,100 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,101 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,116 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,123 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,129 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:25:55,140 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-08-28 15:25:55,141 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,141 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,158 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:55,172 | DEBUG | node.py (2566) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:55,174 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:55,188 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,188 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:55,203 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node4 2017-08-28 15:25:55,204 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:55,205 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,216 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:55,224 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-08-28 15:25:55,225 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:55,236 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,236 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-08-28 15:25:55,243 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-08-28 15:25:55,261 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:55,268 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:25:55,275 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']}, 'Node3') 2017-08-28 15:25:55,288 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:25:55,289 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,289 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,289 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,304 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,304 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,304 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,310 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,323 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None}, 'Node1') 2017-08-28 15:25:55,324 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,332 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,332 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:25:55,334 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,348 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,349 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-08-28 15:25:55,355 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,356 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node3 2017-08-28 15:25:55,366 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,366 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,367 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node3 2017-08-28 15:25:55,377 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,377 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,378 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node1 2017-08-28 15:25:55,387 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,388 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,388 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:25:55,396 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} that it does not need catchup 2017-08-28 15:25:55,397 | DEBUG | node.py (1461) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-08-28 15:25:55,401 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,401 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:55,401 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,402 | DEBUG | node.py (1869) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:55,402 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:25:55,402 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:25:55,414 | INFO | node.py (1471) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:25:55,415 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,421 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:25:55,421 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,422 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-08-28 15:25:55,422 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-08-28 15:25:55,422 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 3 ViewChangeDone messages 2017-08-28 15:25:55,431 | DEBUG | node.py (1550) | caught_up_for_current_view | Node2 does not have view change quorum for view 2 2017-08-28 15:25:55,435 | DEBUG | node.py (1534) | is_catchup_needed | Node2 is not caught up for the current view 2 2017-08-28 15:25:55,435 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,436 | DEBUG | node.py (1539) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-08-28 15:25:55,436 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-08-28 15:25:55,437 | DEBUG | primary_decider.py ( 131) | send | Node2's elector sending VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} 2017-08-28 15:25:55,450 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-08-28 15:25:55,451 | DEBUG | primary_selector.py ( 265) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,470 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node1 2017-08-28 15:25:55,484 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,485 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,506 | DEBUG | node.py (2566) | send | Node2 sending message VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:55,516 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:25:55,517 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node4 2017-08-28 15:25:55,526 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:25:55,527 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node1 2017-08-28 15:25:55,527 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:25:55,537 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"name":"Node3","op":"VIEW_CHANGE_DONE","viewNo":2,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]]}' to Node3 2017-08-28 15:25:55,555 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:55,560 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:55,561 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']}, 'Node4') 2017-08-28 15:25:55,561 | DEBUG | node.py (1254) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:55,561 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,561 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,576 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,583 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,587 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,588 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,599 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node4 2017-08-28 15:25:55,605 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,605 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,605 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:25:55,606 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'txnSeqNo': 15, 'viewNo': 0, 'ledgerId': 1, 'ppSeqNo': 6} from Node4 2017-08-28 15:25:55,614 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,614 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,614 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node4', 'Node1'} that its ledger of type 1 is latest 2017-08-28 15:25:55,640 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:55,648 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,648 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node3', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:25:55,658 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,658 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,659 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 2, 'name': 'Node3'}, 'Node1') 2017-08-28 15:25:55,659 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,672 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node3') 2017-08-28 15:25:55,672 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node1') 2017-08-28 15:25:55,681 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,689 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-08-28 15:25:55,693 | DEBUG | primary_selector.py ( 265) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,694 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,703 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 3) 2017-08-28 15:25:55,704 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node3 2017-08-28 15:25:55,709 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node3 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:25:55,711 | DEBUG | primary_selector.py ( 280) | _startSelection | Node2 starting selection 2017-08-28 15:25:55,718 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node3:0 for instance 0 (view 2) 2017-08-28 15:25:55,718 | INFO | node.py ( 503) | start_participating | Node2 started participating 2017-08-28 15:25:55,719 | DEBUG | replica.py ( 406) | primaryName | Node2:0 setting primaryName for view no 2 to: Node3:0 2017-08-28 15:25:55,719 | DEBUG | replica.py (1678) | _gc | Node2:0 cleaning up till (1, 3) 2017-08-28 15:25:55,727 | DEBUG | replica.py (1693) | _gc | Node2:0 found 3 3-phase keys to clean 2017-08-28 15:25:55,728 | DEBUG | replica.py (1695) | _gc | Node2:0 found 3 request keys to clean 2017-08-28 15:25:55,728 | DEBUG | replica.py ( 311) | h | Node2:0 set watermarks as 0 300 2017-08-28 15:25:55,728 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,734 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node4:1 for instance 1 (view 2) 2017-08-28 15:25:55,734 | DEBUG | replica.py ( 406) | primaryName | Node2:1 setting primaryName for view no 2 to: Node4:1 2017-08-28 15:25:55,739 | DEBUG | replica.py (1678) | _gc | Node2:1 cleaning up till (1, 3) 2017-08-28 15:25:55,740 | DEBUG | replica.py (1693) | _gc | Node2:1 found 3 3-phase keys to clean 2017-08-28 15:25:55,740 | DEBUG | replica.py (1695) | _gc | Node2:1 found 3 request keys to clean 2017-08-28 15:25:55,750 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) from previous checkpoints 2017-08-28 15:25:55,750 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) from previous checkpoints 2017-08-28 15:25:55,751 | DEBUG | replica.py (1716) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) from previous checkpoints 2017-08-28 15:25:55,758 | DEBUG | replica.py ( 311) | h | Node2:1 set watermarks as 0 300 2017-08-28 15:25:55,758 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (1, 3) 2017-08-28 15:25:55,767 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node2:1 set last ordered as (2, 0) 2017-08-28 15:25:55,770 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node2:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,782 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:55,798 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,801 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 2, 'name': 'Node3'}, 'Node4') 2017-08-28 15:25:55,806 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,807 | DEBUG | node.py (1187) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]}, 'Node4') 2017-08-28 15:25:55,822 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} 2017-08-28 15:25:55,823 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node3', 'viewNo': 2, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']]} because it already decided primary which is Node3:0 2017-08-28 15:25:55,866 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 2.207 seconds 2017-08-28 15:25:57,448 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2C got 1 messages through listener 2017-08-28 15:25:57,465 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on request 1503933957374783 2017-08-28 15:25:57,466 | TRACE | node.py (1370) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,467 | DEBUG | node.py (1416) | processClientInBox | Node2C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,481 | DEBUG | node.py (1690) | processRequest | Node2 received client request: SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,491 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,492 | DEBUG | propagator.py ( 148) | propagate | Node2 propagating Rhx2qwaeiGyhU9vn4ynHSS request 1503933957374783 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,493 | DEBUG | node.py (2566) | send | Node2 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:57,502 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since not finalised 2017-08-28 15:25:57,502 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","op":"REQACK"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:57,514 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 98 2017-08-28 15:25:57,515 | TRACE | node.py (1959) | checkPerformance | Node2 checking its performance 2017-08-28 15:25:57,525 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:57,525 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-08-28 15:25:57,525 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:57,526 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:25:57,531 | DEBUG | node.py (1976) | checkPerformance | Node2's master has higher performance than backups 2017-08-28 15:25:57,536 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 100 to run in 10 seconds 2017-08-28 15:25:57,537 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:25:57,538 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node4 2017-08-28 15:25:57,547 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:25:57,553 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node1 2017-08-28 15:25:57,561 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:25:57,562 | TRACE | batched.py ( 95) | flushOutBoxes | Node2 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"data":{"client_port":9702,"alias":"Node5","services":["VALIDATOR"],"node_port":9701,"client_ip":"10.0.0.105","node_ip":"10.0.0.105"},"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc"}}}' to Node3 2017-08-28 15:25:57,576 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 3 messages through listener 2017-08-28 15:25:57,589 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,597 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,598 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}}, 'Node3') 2017-08-28 15:25:57,610 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,616 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,616 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,625 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}}, 'Node4') 2017-08-28 15:25:57,625 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,640 | DEBUG | node.py (2217) | verifySignature | Node2 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,645 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,646 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}}, 'Node1') 2017-08-28 15:25:57,646 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,660 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,661 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,673 | DEBUG | propagator.py ( 213) | forward | Node2 forwarding request ('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783) to 0 replicas 2017-08-28 15:25:57,674 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,684 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,685 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:25:57,692 | DEBUG | node.py (1738) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'reqId': 1503933957374783, 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}, 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS'}} 2017-08-28 15:25:57,692 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} 2017-08-28 15:25:57,705 | DEBUG | propagator.py ( 245) | tryForwarding | Node2 not forwarding request SafeRequest: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}} to its replicas since already forwarded 2017-08-28 15:25:57,764 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.333 seconds 2017-08-28 15:25:57,783 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 2 messages through listener 2017-08-28 15:25:57,793 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node4: PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933957, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:25:57,793 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPREPARE', 'discarded': 1, 'ppTime': 1503933957, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:25:57,793 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933957, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:25:57,794 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933957, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:25:57,794 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'op': 'PREPREPARE', 'ledgerId': 0, 'instId': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933957, 'discarded': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:25:57,794 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933957, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} 2017-08-28 15:25:57,811 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:0 received PRE-PREPARE(2, 1) from Node3:0 at 250253.959767562 2017-08-28 15:25:57,828 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933957, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)], 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:25:57,841 | WARNING | replica.py ( 655) | processReqDuringBatch | Node2:0 encountered exception UnauthorizedClientRequest('Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot add a new node',) while processing Request: {'reqId': 1503933957374783, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'data': {'client_port': 9702, 'alias': 'Node5', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_ip': '10.0.0.105', 'node_ip': '10.0.0.105'}, 'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc'}}, will reject 2017-08-28 15:25:57,842 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(2, 1) at 250253.990819538 2017-08-28 15:25:57,849 | DEBUG | replica.py (2133) | send | Node2:0 sending Prepare 2017-08-28 15:25:57,850 | TRACE | replica.py (2134) | send | Node2:0 sending PREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:25:57,861 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} 2017-08-28 15:25:57,862 | DEBUG | replica.py ( 612) | trackBatches | Node2:0 tracking batch for PREPREPARE{'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'discarded': 0, 'ppTime': 1503933957, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)], 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:57,867 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(2, 1) 2017-08-28 15:25:57,868 | DEBUG | replica.py ( 821) | processPrePrepare | Node2:1 received PRE-PREPARE(2, 1) from Node4:1 at 250254.016850946 2017-08-28 15:25:57,870 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(2, 1) at 250254.018745971 2017-08-28 15:25:57,870 | DEBUG | replica.py (2133) | send | Node2:1 sending Prepare 2017-08-28 15:25:57,870 | TRACE | replica.py (2134) | send | Node2:1 sending PREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:25:57,879 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:25:57,879 | DEBUG | replica.py ( 612) | trackBatches | Node2:1 tracking batch for PREPREPARE{'stateRootHash': None, 'discarded': 1, 'ppTime': 1503933957, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)], 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1} with state root None 2017-08-28 15:25:57,884 | DEBUG | replica.py ( 840) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(2, 1) 2017-08-28 15:25:57,911 | TRACE | zstack.py ( 752) | transmitThroughListener | Node2C transmitting b'{"reqId":1503933957374783,"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","op":"REJECT","reason":"client request invalid: UnauthorizedClientRequest(\'Rhx2qwaeiGyhU9vn4ynHSS is not a steward so cannot add a new node\',)"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:57,918 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'instId': 0, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:57,918 | DEBUG | node.py (2566) | send | Node2 sending message PREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ppSeqNo': 1} to all recipients: ['Node4', 'Node1', 'Node3'] 2017-08-28 15:25:57,919 | TRACE | zstack.py ( 477) | _receiveFromListener | Node2 got 1 messages through listener 2017-08-28 15:25:57,934 | DEBUG | node.py (1239) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:25:57,934 | DEBUG | node.py (1201) | handleOneNodeMsg | Node2 msg validated ({'stateRootHash': None, 'op': 'PREPARE', 'ppTime': 1503933957, 'txnRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:25:57,935 | DEBUG | node.py (1268) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'instId': 1, 'viewNo': 2, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'txnRootHash': None, 'ppSeqNo': 1} 2017-08-28 15:25:57,952 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:57,953 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"viewNo":2,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933957,"digest":"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166"}', b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933957,"digest":"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166"}']) 2017-08-28 15:25:57,966 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node4: b'{"op":"BATCH","messages":["{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"instId\\":0,\\"viewNo\\":2,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166\\"}","{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"viewNo\\":2,\\"txnRootHash\\":null,\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166\\"}"],"signature":null}' 2017-08-28 15:25:57,967 | DEBUG | zstack.py ( 721) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"instId\\":0,\\"viewNo\\":2,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166\\"}","{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"viewNo\\":2,\\"txnRootHash\\":null,\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166\\"}"],"signature":null}' to Node4 2017-08-28 15:25:57,969 | DEBUG | batched.py ( 99) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-08-28 15:25:57,969 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","instId":0,"viewNo":2,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":1,"ppTime":1503933957,"digest":"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166"}', b'{"op":"PREPARE","stateRootHash":null,"instId":1,"viewNo":2,"txnRootHash":null,"ppSeqNo":1,"ppTime":1503933957,"digest":"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166"}']) 2017-08-28 15:25:57,970 | TRACE | batched.py ( 108) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"instId\\":0,\\"viewNo\\":2,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166\\"}","{\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"viewNo\\":2,\\"txnRootHash\\":null,\\"ppSeqNo\\":1,\\"ppTime\\":1503933957,\\"digest\\":\\"8b001054bf64371a96dd