2017-08-28 15:11:49,031 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node1.log 2017-08-28 15:11:49,035 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-08-28 15:11:50,917 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-08-28 15:11:50,940 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-08-28 15:11:51,267 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-08-28 15:11:51,276 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-08-28 15:11:52,563 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-08-28 15:11:52,564 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-08-28 15:11:52,677 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:11:52,699 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:11:52,756 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.05704339701333083 seconds 2017-08-28 15:11:52,812 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node1 2017-08-28 15:11:52,841 | INFO | node.py (2378) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-08-28 15:11:52,929 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:11:52,956 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:11:53,012 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.055551947007188573 seconds 2017-08-28 15:11:53,014 | INFO | node.py (2378) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-08-28 15:11:53,116 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:11:53,118 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:11:53,119 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:11:53,119 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-08-28 15:11:53,119 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-08-28 15:11:53,120 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-08-28 15:11:53,120 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-08-28 15:11:53,122 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-08-28 15:11:53,123 | DEBUG | replica.py ( 311) | h | Node1:0 set watermarks as 0 300 2017-08-28 15:11:53,123 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-08-28 15:11:53,124 | DEBUG | replica.py ( 311) | h | Node1:1 set watermarks as 0 300 2017-08-28 15:11:53,124 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:1 to instance 1 (backup) 2017-08-28 15:11:53,124 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-08-28 15:11:53,124 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-08-28 15:11:53,124 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-08-28 15:11:53,125 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-08-28 15:11:53,125 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-08-28 15:11:53,125 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-08-28 15:11:53,125 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-08-28 15:11:53,125 | DEBUG | node.py ( 337) | __init__ | total plugins loaded in node: 0 2017-08-28 15:11:53,233 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-08-28 15:11:53,260 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-08-28 15:11:53,316 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.056824788014637306 seconds 2017-08-28 15:11:53,345 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node1 has no upgrade events 2017-08-28 15:11:53,347 | INFO | node.py (2378) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-08-28 15:11:53,347 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from stopped to starting 2017-08-28 15:11:53,347 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:11:53,348 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:11:53,348 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-08-28 15:11:53,348 | DEBUG | zstack.py ( 319) | start | Node1 starting with restricted as True and reSetupAuth as True 2017-08-28 15:11:53,349 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-08-28 15:11:53,350 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:11:53,350 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node1/public_keys] 2017-08-28 15:11:53,351 | DEBUG | zstack.py ( 347) | open | Node1 will bind its listener at 9701 2017-08-28 15:11:53,352 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-08-28 15:11:53,352 | DEBUG | zstack.py ( 319) | start | Node1C starting with restricted as False and reSetupAuth as True 2017-08-28 15:11:53,352 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-08-28 15:11:53,352 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-08-28 15:11:53,353 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-08-28 15:11:53,353 | DEBUG | zstack.py ( 347) | open | Node1C will bind its listener at 9702 2017-08-28 15:11:53,354 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-08-28 15:11:53,355 | INFO | node.py ( 612) | start | Node1 first time running... 2017-08-28 15:11:53,359 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node1 found the following missing connections: Node2, Node3, Node4 2017-08-28 15:11:53,360 | TRACE | remote.py ( 84) | connect | connecting socket 78 44490496 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-08-28 15:11:53,360 | INFO | zstack.py ( 588) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-08-28 15:11:53,361 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node2 2017-08-28 15:11:53,361 | TRACE | remote.py ( 84) | connect | connecting socket 79 44595184 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:11:53,362 | INFO | zstack.py ( 588) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-08-28 15:11:53,363 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node3 2017-08-28 15:11:53,364 | TRACE | remote.py ( 84) | connect | connecting socket 80 44632464 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:11:53,364 | INFO | zstack.py ( 588) | connect | CONNECTION: Node1 looking for Node4 at 10.0.0.5:9707 2017-08-28 15:11:53,364 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node4 2017-08-28 15:11:53,364 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:11:53,383 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node2 2017-08-28 15:11:53,384 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node2 2017-08-28 15:11:53,393 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node3 2017-08-28 15:11:53,393 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-08-28 15:11:53,394 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node4 2017-08-28 15:11:53,394 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node4 2017-08-28 15:12:03,126 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-08-28 15:12:03,127 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:03,127 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-08-28 15:12:08,366 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:08,367 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:08,367 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:08,368 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node2 2017-08-28 15:12:08,368 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node3 2017-08-28 15:12:08,368 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node4 2017-08-28 15:12:08,369 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:12:08,382 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node2 2017-08-28 15:12:08,383 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node2 2017-08-28 15:12:08,383 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node3 2017-08-28 15:12:08,383 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-08-28 15:12:08,384 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node4 2017-08-28 15:12:08,384 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node4 2017-08-28 15:12:13,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 4 2017-08-28 15:12:13,135 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:13,135 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-08-28 15:12:15,323 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:15,324 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from Node2 2017-08-28 15:12:15,324 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged Node2 2017-08-28 15:12:15,325 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'po' to Node2 2017-08-28 15:12:15,327 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'po' to Node2 2017-08-28 15:12:15,403 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:15,404 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node2 2017-08-28 15:12:15,404 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node2 2017-08-28 15:12:15,405 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from set() to {'Node2'} 2017-08-28 15:12:15,407 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-08-28 15:12:15,408 | DEBUG | node.py (2566) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node2'] 2017-08-28 15:12:15,413 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}' to Node2 2017-08-28 15:12:15,414 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}' to Node2 2017-08-28 15:12:15,428 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:15,429 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:15,431 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None}, 'Node2') 2017-08-28 15:12:15,432 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:15,435 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node2 2017-08-28 15:12:15,436 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:15,436 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:23,142 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 5 2017-08-28 15:12:23,143 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:23,143 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-08-28 15:12:23,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:23,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:23,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:23,369 | DEBUG | zstack.py ( 603) | reconnectRemote | Node1 reconnecting to Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:23,370 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:23,371 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 79 44595184 2017-08-28 15:12:23,371 | TRACE | remote.py ( 84) | connect | connecting socket 79 44594944 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:23,372 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node3 2017-08-28 15:12:23,372 | DEBUG | zstack.py ( 603) | reconnectRemote | Node1 reconnecting to Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:23,372 | DEBUG | remote.py ( 87) | disconnect | disconnecting remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:23,372 | TRACE | remote.py ( 90) | disconnect | disconnecting socket 80 44632464 2017-08-28 15:12:23,376 | TRACE | remote.py ( 84) | connect | connecting socket 80 44632032 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:23,376 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node4 2017-08-28 15:12:23,377 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:12:23,391 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node3 2017-08-28 15:12:23,393 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-08-28 15:12:23,393 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node4 2017-08-28 15:12:23,394 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node4 2017-08-28 15:12:33,154 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 6 2017-08-28 15:12:33,156 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:33,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-08-28 15:12:38,372 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:38,373 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:38,373 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:38,374 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node3 2017-08-28 15:12:38,374 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 pinged Node4 2017-08-28 15:12:38,374 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:12:38,398 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node3 2017-08-28 15:12:38,399 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-08-28 15:12:38,402 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'pi' to Node4 2017-08-28 15:12:38,403 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'pi' to Node4 2017-08-28 15:12:43,172 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-08-28 15:12:43,173 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:43,173 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-08-28 15:12:43,596 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:12:43,597 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from Node3 2017-08-28 15:12:43,597 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged Node3 2017-08-28 15:12:43,604 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node3 2017-08-28 15:12:43,604 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node3 2017-08-28 15:12:43,609 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'po' to Node3 2017-08-28 15:12:43,610 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'po' to Node3 2017-08-28 15:12:43,611 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2'} to {'Node2', 'Node3'} 2017-08-28 15:12:43,615 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-08-28 15:12:43,616 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from starting to started_hungry 2017-08-28 15:12:43,616 | DEBUG | node.py ( 923) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node3'} 2017-08-28 15:12:43,616 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-08-28 15:12:43,617 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-08-28 15:12:43,617 | DEBUG | node.py (2566) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,618 | DEBUG | node.py (2566) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,630 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,640 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[],"viewNo":0}', b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}']) 2017-08-28 15:12:43,640 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"ledgerId\\":0,\\"viewNo\\":null}"],"op":"BATCH"}' 2017-08-28 15:12:43,643 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"ledgerId\\":0,\\"viewNo\\":null}"],"op":"BATCH"}' to Node3 2017-08-28 15:12:43,698 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:43,699 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:43,699 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'LEDGER_STATUS', 'viewNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ledgerId': 0, 'ppSeqNo': None}, 'Node3') 2017-08-28 15:12:43,699 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:43,700 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node3 2017-08-28 15:12:43,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,708 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,712 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:12:43,712 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-08-28 15:12:43,712 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,713 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,713 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,713 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,713 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,714 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:12:43,718 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:12:43,719 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:12:43,719 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:43,719 | DEBUG | node.py ( 923) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node3'} 2017-08-28 15:12:43,719 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:12:43,725 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-08-28 15:12:43,725 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,725 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-08-28 15:12:43,725 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,725 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-08-28 15:12:43,726 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,726 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 2 2017-08-28 15:12:43,726 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:12:43,726 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:12:43,733 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,733 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,733 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:12:43,739 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:12:43,740 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,740 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,740 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:12:43,743 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:12:43,744 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,744 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,745 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:12:43,745 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node4 2017-08-28 15:12:43,745 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,787 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:43,792 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,792 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:12:43,792 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,793 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,793 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:12:43,793 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,793 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,793 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:12:43,793 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,794 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,795 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,812 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,812 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,813 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:12:43,819 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:12:43,825 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:12:43,830 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,830 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:12:43,830 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:12:43,831 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,831 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:12:43,831 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,831 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,831 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:12:43,839 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,840 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,840 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:12:43,840 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:43,840 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,841 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,841 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,841 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,841 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,841 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,842 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:12:43,842 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,843 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,858 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:12:43,858 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,858 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,859 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,865 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,866 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,867 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:12:43,867 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:12:43,876 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:12:43,883 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:43,885 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:12:43,886 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:12:43,886 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:12:43,886 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,886 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,886 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,886 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,887 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:12:43,887 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:43,895 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:12:43,895 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,895 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,895 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:12:43,904 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-08-28 15:12:43,904 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:12:43,905 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,905 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:43,905 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:43,905 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:43,905 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:12:43,905 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:12:43,906 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:12:43,906 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,906 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:12:43,906 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-08-28 15:12:43,906 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:12:43,907 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-08-28 15:12:43,907 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:12:43,921 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-08-28 15:12:43,921 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:43,921 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-08-28 15:12:43,921 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:43,921 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 1 2017-08-28 15:12:43,922 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:12:43,922 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-08-28 15:12:43,922 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:12:43,923 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:12:43,923 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,923 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:43,937 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:43,937 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,938 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:43,939 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:12:43,939 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:43,940 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,940 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:43,947 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:12:43,947 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:12:43,948 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:43,948 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:43,948 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:12:43,948 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:43,965 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.179 seconds 2017-08-28 15:12:43,978 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:43,978 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:43,979 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:12:43,979 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:43,987 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,989 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:12:43,989 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,989 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:43,990 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:12:43,990 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,020 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:12:44,025 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:12:44,046 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:12:44,047 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:44,047 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:44,052 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:12:44,054 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:12:44,054 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:44,054 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:12:44,065 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:12:44,065 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,065 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,066 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,066 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,066 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,075 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,076 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:44,076 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:12:44,076 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:12:44,076 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,076 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:12:44,076 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,077 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,077 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:12:44,077 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,087 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:44,088 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:12:44,088 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node2 2017-08-28 15:12:44,089 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,089 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,089 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node2 2017-08-28 15:12:44,089 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,090 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,104 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:12:44,105 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:44,105 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:44,105 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:12:44,107 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:44,111 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']} 2017-08-28 15:12:44,111 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:12:44,111 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}']} 2017-08-28 15:12:44,112 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,112 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,112 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,112 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,112 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:12:44,112 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:44,113 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node3 2017-08-28 15:12:44,113 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,113 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,113 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:12:44,114 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} that it does not need catchup 2017-08-28 15:12:44,114 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:12:44,114 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,114 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:12:44,114 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:12:44,114 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:12:44,114 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:12:44,114 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:12:44,125 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:12:44,126 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,126 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:12:44,126 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,126 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:12:44,126 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:12:44,126 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 2 ViewChangeDone messages 2017-08-28 15:12:44,127 | DEBUG | node.py (1550) | caught_up_for_current_view | Node1 does not have view change quorum for view 0 2017-08-28 15:12:44,127 | DEBUG | node.py (1534) | is_catchup_needed | Node1 is not caught up for the current view 0 2017-08-28 15:12:44,127 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:12:44,133 | DEBUG | node.py (1539) | is_catchup_needed | Node1 ordered till last prepared certificate 2017-08-28 15:12:44,134 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-08-28 15:12:44,134 | DEBUG | primary_decider.py ( 131) | send | Node1's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,134 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 1 ViewChangeDone messages 2017-08-28 15:12:44,135 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:12:44,143 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node3 2017-08-28 15:12:44,144 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,144 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:44,145 | DEBUG | node.py (2566) | send | Node1 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:12:44,162 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node2 2017-08-28 15:12:44,162 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node2 2017-08-28 15:12:44,167 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node3 2017-08-28 15:12:44,168 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node3 2017-08-28 15:12:44,168 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node4 2017-08-28 15:12:44,168 | DEBUG | zstack.py ( 726) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-08-28 15:12:44,168 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node1","viewNo":0}' to Node4 2017-08-28 15:12:44,193 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:12:44,194 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,194 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'viewNo': 0}, 'Node2') 2017-08-28 15:12:44,195 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,195 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,205 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-08-28 15:12:44,205 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,206 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node2') 2017-08-28 15:12:44,207 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-08-28 15:12:44,259 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,260 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (2 >= 2) 2017-08-28 15:12:44,260 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node1 2017-08-28 15:12:44,260 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:12:44,261 | DEBUG | primary_selector.py ( 280) | _startSelection | Node1 starting selection 2017-08-28 15:12:44,261 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 0) 2017-08-28 15:12:44,261 | INFO | node.py ( 503) | start_participating | Node1 started participating 2017-08-28 15:12:44,261 | DEBUG | replica.py ( 406) | primaryName | Node1:0 setting primaryName for view no 0 to: Node1:0 2017-08-28 15:12:44,261 | DEBUG | replica.py (1678) | _gc | Node1:0 cleaning up till (0, 0) 2017-08-28 15:12:44,267 | DEBUG | replica.py (1693) | _gc | Node1:0 found 0 3-phase keys to clean 2017-08-28 15:12:44,267 | DEBUG | replica.py (1695) | _gc | Node1:0 found 0 request keys to clean 2017-08-28 15:12:44,267 | DEBUG | replica.py ( 311) | h | Node1:0 set watermarks as 0 300 2017-08-28 15:12:44,268 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,268 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:1 selected primary Node2:1 for instance 1 (view 0) 2017-08-28 15:12:44,268 | DEBUG | replica.py ( 406) | primaryName | Node1:1 setting primaryName for view no 0 to: Node2:1 2017-08-28 15:12:44,268 | DEBUG | replica.py (1678) | _gc | Node1:1 cleaning up till (0, 0) 2017-08-28 15:12:44,268 | DEBUG | replica.py (1693) | _gc | Node1:1 found 0 3-phase keys to clean 2017-08-28 15:12:44,269 | DEBUG | replica.py (1695) | _gc | Node1:1 found 0 request keys to clean 2017-08-28 15:12:44,269 | DEBUG | replica.py ( 311) | h | Node1:1 set watermarks as 0 300 2017-08-28 15:12:44,269 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node1:1 Setting last ordered for non-master as (0, 0) 2017-08-28 15:12:44,269 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 0) 2017-08-28 15:12:44,269 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:12:44,281 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:44,282 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-08-28 15:12:44,287 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.311 seconds 2017-08-28 15:12:52,068 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:52,071 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from Node4 2017-08-28 15:12:52,072 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged Node4 2017-08-28 15:12:52,078 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'po' to Node4 2017-08-28 15:12:52,079 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'po' to Node4 2017-08-28 15:12:52,172 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:52,173 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node4 2017-08-28 15:12:52,175 | DEBUG | zstack.py ( 667) | handlePingPong | Node1 got pong from Node4 2017-08-28 15:12:52,177 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:12:52,178 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:12:52,187 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:12:52,188 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,189 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,189 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,190 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,190 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,190 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,190 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,191 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,191 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,194 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,195 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,196 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 9, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 1, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,196 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:12:52,196 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,197 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,197 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:12:52,197 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:12:52,197 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,197 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:12:52,198 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:12:52,198 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node4 2017-08-28 15:12:52,211 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,211 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,211 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:12:52,215 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} from Node4 2017-08-28 15:12:52,215 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,218 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 9 with 9 2017-08-28 15:12:52,219 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:12:52,234 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2', 'Node3'} to {'Node2', 'Node3', 'Node4'} 2017-08-28 15:12:52,239 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-08-28 15:12:52,240 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started_hungry to started 2017-08-28 15:12:52,244 | DEBUG | node.py ( 923) | checkInstances | Node1 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node3', 'Node4'} 2017-08-28 15:12:52,244 | DEBUG | node.py ( 884) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to lagged node Node4 2017-08-28 15:12:52,246 | DEBUG | node.py (2566) | send | Node1 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,247 | DEBUG | node.py (2566) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,255 | DEBUG | node.py (2566) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,258 | DEBUG | node.py (2566) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,265 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,281 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0}', b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}', b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","txnSeqNo":9,"ledgerId":1,"viewNo":null}', b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2,"viewNo":null}']) 2017-08-28 15:12:52,282 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"ledgerId\\":0,\\"viewNo\\":null}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"ledgerId\\":1,\\"viewNo\\":null}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"ledgerId\\":2,\\"viewNo\\":null}"],"op":"BATCH"}' 2017-08-28 15:12:52,292 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,9,\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"ledgerId\\":0,\\"viewNo\\":null}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"txnSeqNo\\":9,\\"ledgerId\\":1,\\"viewNo\\":null}","{\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"ledgerId\\":2,\\"viewNo\\":null}"],"op":"BATCH"}' to Node4 2017-08-28 15:12:52,301 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.131 seconds 2017-08-28 15:12:52,477 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:52,477 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:12:52,478 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:12:52,478 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"op":"LEDGER_STATUS"}']} 2017-08-28 15:12:52,493 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:52,494 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,494 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-08-28 15:12:52,495 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:52,499 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None}, 'Node4') 2017-08-28 15:12:52,499 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:52,501 | DEBUG | node.py ( 889) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node4 2017-08-28 15:12:52,502 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node4 2017-08-28 15:12:52,502 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,502 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,520 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:12:52,520 | DEBUG | zstack.py ( 661) | handlePingPong | Node1C got ping from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,521 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'po' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,521 | DEBUG | zstack.py ( 643) | sendPingPong | Node1C ponged b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,563 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:12:52,564 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:52,573 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} 2017-08-28 15:12:52,574 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:12:52,580 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,589 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"ledgerId":0,"viewNo":null}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:12:52,589 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,597 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:12:52,645 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.170 seconds 2017-08-28 15:12:52,839 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:12:52,846 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-08-28 15:12:52,851 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:12:52,858 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,9,"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-08-28 15:12:52,863 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,863 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,864 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,864 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,865 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:12:52,868 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:12:52,868 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,869 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,869 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,875 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,875 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:12:52,876 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:12:52,885 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:52,885 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node1', 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,885 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:52,886 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,892 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,897 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,898 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 9, 'ppSeqNo': None, 'merkleRoot': '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t', 'ledgerId': 1, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:12:52,904 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node4') 2017-08-28 15:12:52,915 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-08-28 15:12:52,916 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 9, '42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-08-28 15:12:52,919 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 4 msgs to Node4 into one transmission 2017-08-28 15:12:52,920 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:12:52,920 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:12:52,924 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"42dQyZ5xFGCTdN3EEkEkAGKoyZv7NGsBF1ps6DZ1KM1t\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:12:53,127 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 2 2017-08-28 15:12:53,128 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:12:53,128 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:12:53,128 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 9 to run in 60 seconds 2017-08-28 15:12:53,139 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-08-28 15:12:53,140 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:12:53,140 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-08-28 15:12:53,180 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 8 2017-08-28 15:12:53,181 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:12:53,181 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:12:53,181 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:12:53,182 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:12:53,182 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:12:53,182 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:12:53,183 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-08-28 15:12:53,360 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action propose_view_change with id 3 2017-08-28 15:12:53,361 | DEBUG | throttler.py ( 31) | acquire | now: 249469.510652536, len(actionsLog): 0 2017-08-28 15:12:53,362 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:12:53,363 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:12:53,364 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-08-28 15:12:53,364 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249409.272495385, 249409.27303626] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-08-28 15:12:53,367 | DEBUG | node.py (2566) | send | Node1 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:12:53,368 | DEBUG | node.py (1939) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-08-28 15:12:53,368 | INFO | node.py (2089) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-08-28 15:12:53,369 | DEBUG | node.py (1939) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-08-28 15:12:53,370 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2 2017-08-28 15:12:53,370 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2 2017-08-28 15:12:53,370 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-08-28 15:12:53,371 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-08-28 15:12:53,371 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node4 2017-08-28 15:12:53,372 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node4 2017-08-28 15:12:53,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:12:53,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:12:53,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:12:53,387 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:13:03,187 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 10 2017-08-28 15:13:03,188 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:03,188 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:03,193 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:13:03,193 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:03,194 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:03,194 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:03,196 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-08-28 15:13:08,395 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:08,395 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:08,395 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:08,396 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:13:13,210 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 11 2017-08-28 15:13:13,210 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:13,210 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:13,211 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:13:13,211 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:13,211 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:13,212 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:13,212 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-08-28 15:13:20,674 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:13:20,677 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933200605368 2017-08-28 15:13:20,689 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,699 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,701 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,702 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,705 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933200605368 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:13:20,706 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:13:20,706 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since not finalised 2017-08-28 15:13:20,713 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:20,717 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node2 2017-08-28 15:13:20,717 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node2 2017-08-28 15:13:20,723 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node3 2017-08-28 15:13:20,724 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node3 2017-08-28 15:13:20,730 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node4 2017-08-28 15:13:20,731 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933200605368}}' to Node4 2017-08-28 15:13:20,737 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:13:20,746 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,750 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,751 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}}, 'Node4') 2017-08-28 15:13:20,754 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,758 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,758 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,758 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}}, 'Node3') 2017-08-28 15:13:20,759 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,765 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933200605368 2017-08-28 15:13:20,771 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,771 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}}, 'Node2') 2017-08-28 15:13:20,772 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,772 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,778 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,779 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to 0 replicas 2017-08-28 15:13:20,781 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'reqId': 1503933200605368}} 2017-08-28 15:13:20,781 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,782 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since already forwarded 2017-08-28 15:13:20,782 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368}} 2017-08-28 15:13:20,783 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} 2017-08-28 15:13:20,787 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn', 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933200605368} to its replicas since already forwarded 2017-08-28 15:13:20,795 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 1 for ledger 1 with state root b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[' 2017-08-28 15:13:20,835 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:13:20,835 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:13:20,836 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 1} 2017-08-28 15:13:20,836 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 1} with state root b'Q;\xde\xde\xb7\xfdA\xdf\x99 T\x7fe2Z\xbe\xf5v)(\xb8-\xd2\x19"\x82\xa2d\xa0h\xac[' 2017-08-28 15:13:20,840 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:13:20,848 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 1} 2017-08-28 15:13:20,848 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'ppTime': 1503933200, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:13:20,853 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933200605368]], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 1} 2017-08-28 15:13:20,862 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933200, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:13:20,871 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 1) from Node2:1 at 249497.020530513 2017-08-28 15:13:20,872 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 1) at 249497.021232922 2017-08-28 15:13:20,872 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:13:20,875 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,876 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,876 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933200, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 1} with state root None 2017-08-28 15:13:20,877 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 1) 2017-08-28 15:13:20,895 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:13:20,896 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:13:20,896 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:13:20,896 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:13:20,897 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:13:20,897 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933200605368]],"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ledgerId":1,"ppTime":1503933200,"ppSeqNo":1,"instId":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:13:20,898 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:13:20,917 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:13:20,923 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:13:20,923 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:13:20,926 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:13:20,926 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:13:20,926 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","op":"PREPARE","stateRootHash":null,"ppTime":1503933200,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:13:20,937 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:13:20,943 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,944 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:20,944 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,944 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,947 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933200, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:20,948 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,951 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 1) from Node3:1 2017-08-28 15:13:20,965 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 1) at 249497.113979969 2017-08-28 15:13:20,965 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:13:20,965 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:20,970 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:13:20,971 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:20,971 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 1) from Node4:1 2017-08-28 15:13:20,977 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:20,977 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 1) 2017-08-28 15:13:20,990 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:13:21,007 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node2 2017-08-28 15:13:21,008 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node2 2017-08-28 15:13:21,008 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,013 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,018 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,019 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,028 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:13:21,029 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,034 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933200, 'op': 'PREPARE', 'ppSeqNo': 1, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0}, 'Node2') 2017-08-28 15:13:21,034 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,051 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 1) from Node2:0 2017-08-28 15:13:21,052 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,052 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,057 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:13:21,058 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,064 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:13:21,064 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,067 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 1) from Node2:1 2017-08-28 15:13:21,082 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:13:21,082 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,083 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:13:21,089 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:13:21,090 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:13:21,090 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd","ppSeqNo":1,"viewNo":0,"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849"}', '{"ppSeqNo":1,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:13:21,097 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,098 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933200, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,098 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,105 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,105 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,105 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,106 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']} 2017-08-28 15:13:21,107 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:13:21,112 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","viewNo":0,"op":"PREPARE","ppSeqNo":1,"instId":0,"digest":"3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849","ppTime":1503933200,"stateRootHash":"HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd"}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":1,"instId":1}']} 2017-08-28 15:13:21,112 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,112 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:13:21,121 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,121 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,121 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node3') 2017-08-28 15:13:21,122 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} 2017-08-28 15:13:21,134 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 1) from Node4:0 2017-08-28 15:13:21,141 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 1) at 249497.290322347 2017-08-28 15:13:21,141 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:13:21,142 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,146 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:13:21,146 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,147 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 1) from Node3:0 2017-08-28 15:13:21,153 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '3c386236a4ea8c369ff62606d4bfc41472e042bc405eb87edb4aa6741d91a849', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ppTime': 1503933200, 'ppSeqNo': 1} 2017-08-28 15:13:21,153 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 1) 2017-08-28 15:13:21,153 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 1) from Node4:1 2017-08-28 15:13:21,154 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:13:21,154 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 1) 2017-08-28 15:13:21,154 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 1) 2017-08-28 15:13:21,161 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:13:21,162 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 1, 'ppTime': 1503933200, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:13:21,162 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 1) 2017-08-28 15:13:21,167 | DEBUG | replica.py (1540) | _newCheckpointState | Node1:1 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,167 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,168 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:13:21,186 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:13:21,187 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:13:21,190 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:13:21,191 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,197 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:13:21,197 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,197 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,204 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node3') 2017-08-28 15:13:21,204 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,214 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node2 2017-08-28 15:13:21,214 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node2 2017-08-28 15:13:21,215 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,221 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-08-28 15:13:21,221 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,222 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-08-28 15:13:21,232 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 1) from Node4:0 2017-08-28 15:13:21,233 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:13:21,235 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,236 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 1) from Node3:0 2017-08-28 15:13:21,236 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:13:21,236 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 1) 2017-08-28 15:13:21,236 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 1) 2017-08-28 15:13:21,237 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:13:21,244 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 1, 'ppTime': 1503933200, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368)], 'stateRootHash': 'HXWtnybPGscLBAZk2QjGHGLeNoXXDLvwgzk8LRK1MmZd', 'txnRootHash': 'ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:13:21,244 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 1) 2017-08-28 15:13:21,244 | DEBUG | replica.py (1540) | _newCheckpointState | Node1:0 adding new checkpoint state for (1, 100) 2017-08-28 15:13:21,245 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 1) 2017-08-28 15:13:21,246 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:13:21,256 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,256 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:13:21,256 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} 2017-08-28 15:13:21,275 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 1 of 1 requests 2017-08-28 15:13:21,292 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:13:21,322 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) to client 2017-08-28 15:13:21,323 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"4GMeDHCT3dxL3amjDEFBWCZrAmCS3foqhnpwdsML4CXuHYFNm5yKiRBZn9dqoyLKV8CQpCc7mwAyVZdKsxyUp2zn","txnTime":1503933200,"auditPath":["jGLMZFGcgHUKKu3WgPHBiDkc9H9CVvA4gmrmS3PnmUb","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe61","seqNo":10,"type":"1","rootHash":"ADLAyBWGfPdPCc49Tw3LVD2t1sct2skszLn9XUR6PayE","reqId":1503933200605368},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:13:21,324 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 1) for ledger 1 range (10, 10) 2017-08-28 15:13:21,324 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:21,325 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 1, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:13:21,353 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.681 seconds 2017-08-28 15:13:23,228 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 12 2017-08-28 15:13:23,228 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:23,229 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:23,229 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:13:23,232 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:23,233 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:23,233 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:23,234 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-08-28 15:13:23,410 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:23,419 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:23,420 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:23,420 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:13:33,244 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 13 2017-08-28 15:13:33,245 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:33,245 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:33,246 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:13:33,246 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:33,246 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:33,247 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:33,247 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-08-28 15:13:38,420 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:38,426 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:38,427 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:38,440 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:13:43,247 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 14 2017-08-28 15:13:43,248 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:43,248 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:43,249 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:13:43,251 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:43,253 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:43,253 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:43,253 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-08-28 15:13:53,151 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 9 2017-08-28 15:13:53,152 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:13:53,152 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:53,152 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 16 to run in 60 seconds 2017-08-28 15:13:53,152 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 2 2017-08-28 15:13:53,153 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:13:53,168 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-08-28 15:13:53,274 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 15 2017-08-28 15:13:53,279 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:13:53,281 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:13:53,283 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:13:53,284 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:13:53,284 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:13:53,287 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:13:53,288 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-08-28 15:13:53,431 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:13:53,434 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:13:53,436 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:13:53,437 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:14:03,289 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 17 2017-08-28 15:14:03,296 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:03,297 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:03,297 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:03,297 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:03,305 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:03,305 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:03,306 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-08-28 15:14:08,453 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:08,461 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:08,461 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:08,462 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:14:13,333 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 18 2017-08-28 15:14:13,333 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:13,334 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:13,334 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:13,334 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:13,335 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:13,342 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:13,342 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-08-28 15:14:23,369 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 19 2017-08-28 15:14:23,369 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:23,370 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:23,371 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:23,371 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:23,374 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:23,375 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:23,375 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-08-28 15:14:23,459 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:23,459 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:23,460 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:23,471 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:14:33,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 20 2017-08-28 15:14:33,392 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:33,392 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:33,392 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:33,393 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:33,393 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:33,393 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:33,395 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-08-28 15:14:38,484 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:38,493 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:38,494 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:38,494 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:14:43,417 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 21 2017-08-28 15:14:43,423 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:43,426 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:43,426 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:43,426 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:43,427 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:43,427 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:43,429 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-08-28 15:14:53,158 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 16 2017-08-28 15:14:53,162 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:14:53,163 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:53,166 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 23 to run in 60 seconds 2017-08-28 15:14:53,169 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 3 2017-08-28 15:14:53,179 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:14:53,180 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-08-28 15:14:53,453 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 22 2017-08-28 15:14:53,455 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:14:53,459 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:14:53,463 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:14:53,463 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:14:53,464 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:14:53,464 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:14:53,464 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-08-28 15:14:53,494 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:14:53,498 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:14:53,498 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:14:53,499 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:15:03,473 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 24 2017-08-28 15:15:03,480 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:03,480 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:03,480 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:03,484 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:03,484 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:03,484 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:03,486 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-08-28 15:15:08,505 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:08,512 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:08,515 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:08,516 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:15:13,496 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 25 2017-08-28 15:15:13,501 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:13,501 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:13,503 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:13,503 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:13,504 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:13,507 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:13,512 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-08-28 15:15:23,518 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 26 2017-08-28 15:15:23,527 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:23,535 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:23,536 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:23,537 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:23,545 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:23,547 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:23,552 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-08-28 15:15:23,558 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:23,558 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:23,558 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:23,559 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:15:33,555 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 27 2017-08-28 15:15:33,559 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:33,562 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:33,564 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:33,570 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:33,571 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:33,571 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:33,571 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-08-28 15:15:38,577 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:38,582 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:38,584 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:38,584 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:15:43,602 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 28 2017-08-28 15:15:43,610 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:43,611 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:43,612 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:43,613 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:43,613 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:43,613 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:43,614 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-08-28 15:15:53,190 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 23 2017-08-28 15:15:53,191 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:15:53,192 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:53,193 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 30 to run in 60 seconds 2017-08-28 15:15:53,196 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 4 2017-08-28 15:15:53,196 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:15:53,203 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-08-28 15:15:53,585 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:15:53,596 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:15:53,597 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:15:53,598 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:15:53,627 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 29 2017-08-28 15:15:53,628 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:15:53,629 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:15:53,629 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:15:53,630 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:15:53,630 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:15:53,630 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:15:53,631 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-08-28 15:16:03,637 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 31 2017-08-28 15:16:03,639 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:03,640 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:03,641 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:03,641 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:03,641 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:03,641 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:03,643 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-08-28 15:16:08,601 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:08,603 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:08,604 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:08,604 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:16:13,656 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 32 2017-08-28 15:16:13,656 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:13,657 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:13,657 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:13,657 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:13,661 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:13,662 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:13,662 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-08-28 15:16:23,613 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:23,623 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:23,623 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:23,624 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:16:23,674 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 33 2017-08-28 15:16:23,674 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:23,675 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:23,675 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:23,675 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:23,688 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:23,689 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:23,689 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-08-28 15:16:33,699 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 34 2017-08-28 15:16:33,700 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:33,700 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:33,700 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:33,700 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:33,703 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:33,708 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:33,715 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 35 to run in 10 seconds 2017-08-28 15:16:38,618 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:38,638 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:38,639 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:38,650 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:16:43,755 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 35 2017-08-28 15:16:43,756 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:43,756 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:43,756 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:43,756 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:43,757 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:43,757 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:43,757 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-08-28 15:16:53,212 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 30 2017-08-28 15:16:53,231 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:16:53,232 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:53,232 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 37 to run in 60 seconds 2017-08-28 15:16:53,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 5 2017-08-28 15:16:53,233 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:16:53,233 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-08-28 15:16:53,653 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:16:53,669 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:16:53,670 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:16:53,670 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:16:53,758 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 36 2017-08-28 15:16:53,763 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:16:53,764 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:16:53,766 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:16:53,766 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:16:53,767 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:16:53,774 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:16:53,774 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-08-28 15:16:57,864 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:16:57,865 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:57,867 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:16:57,888 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:16:57,888 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:03,782 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 38 2017-08-28 15:17:03,789 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:03,789 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:03,789 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:17:03,790 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:03,795 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:03,796 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:03,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-08-28 15:17:07,438 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:07,438 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,439 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:07,455 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:07,459 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:08,687 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:08,688 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:08,688 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:08,689 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:17:12,653 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:12,659 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,662 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:12,677 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:12,686 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:13,797 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 39 2017-08-28 15:17:13,807 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:13,809 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:13,810 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:17:13,813 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:13,817 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:13,820 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:13,820 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-08-28 15:17:22,303 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:22,304 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,308 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:22,322 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:22,332 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:23,695 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:23,695 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:23,710 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:23,711 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:17:23,832 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 40 2017-08-28 15:17:23,833 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:23,836 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:23,837 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:17:23,838 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:23,838 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:23,839 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:23,839 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-08-28 15:17:27,904 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:27,908 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:27,912 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:27,938 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:27,938 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:33,857 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 41 2017-08-28 15:17:33,863 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:33,864 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:33,878 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.751796821719468 is acceptable. 2017-08-28 15:17:33,882 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:33,882 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:33,883 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:33,883 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-08-28 15:17:37,543 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:37,544 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,547 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:37,558 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:37,571 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:38,719 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:38,730 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:38,730 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:38,736 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:17:39,554 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:17:39,588 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933459424488 2017-08-28 15:17:39,588 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,589 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,589 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,599 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,600 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933459424488 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:39,600 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:39,602 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since not finalised 2017-08-28 15:17:39,603 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:39,624 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node2 2017-08-28 15:17:39,624 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node2 2017-08-28 15:17:39,625 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node3 2017-08-28 15:17:39,625 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node3 2017-08-28 15:17:39,636 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node4 2017-08-28 15:17:39,637 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","operation":{"type":"1","role":"2","dest":"XhYtvJqezMUKfF6KVNaGmT","verkey":"~RmCt3RtDvKDfQBSKxo4qvy"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933459424488}}' to Node4 2017-08-28 15:17:39,638 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:39,663 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,664 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,667 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}}, 'Node2') 2017-08-28 15:17:39,668 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,668 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,683 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,684 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}}, 'Node4') 2017-08-28 15:17:39,684 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,685 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488}} 2017-08-28 15:17:39,693 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,694 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to 0 replicas 2017-08-28 15:17:39,695 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,703 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,706 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since already forwarded 2017-08-28 15:17:39,717 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 2 for ledger 1 with state root b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t' 2017-08-28 15:17:39,739 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:39,765 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:17:39,765 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 1} 2017-08-28 15:17:39,766 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 1} with state root b'\xf5\x8c7\x16\xc3\xc8\x11\xf8J\x95Y\xfe\xc1\xf2\x981c\xfa\x19\x1b\xe7\xab\xd8\x0b\xed\x9a9z\xdb\x13\x98t' 2017-08-28 15:17:39,768 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:39,777 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933459424488 2017-08-28 15:17:39,782 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,790 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}}, 'Node3') 2017-08-28 15:17:39,790 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,791 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'reqId': 1503933459424488}} 2017-08-28 15:17:39,793 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} 2017-08-28 15:17:39,797 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe', 'operation': {'type': '1', 'role': '2', 'dest': 'XhYtvJqezMUKfF6KVNaGmT', 'verkey': '~RmCt3RtDvKDfQBSKxo4qvy'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933459424488} to its replicas since already forwarded 2017-08-28 15:17:39,818 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933459, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:39,829 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:39,832 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 1} 2017-08-28 15:17:39,848 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'ppTime': 1503933459, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:17:39,848 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933459424488]], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 1} 2017-08-28 15:17:39,867 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:39,870 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:39,878 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:39,878 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:39,888 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:39,889 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933459424488]],"stateRootHash":"6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR","ledgerId":1,"ppTime":1503933459,"ppSeqNo":2,"instId":0,"txnRootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:39,900 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 2) from Node2:1 at 249756.049261717 2017-08-28 15:17:39,901 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 2) at 249756.049932473 2017-08-28 15:17:39,916 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:17:39,916 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:39,916 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:39,917 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933459, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 1} with state root None 2017-08-28 15:17:39,917 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 2) 2017-08-28 15:17:39,949 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:39,982 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:39,982 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:39,984 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:39,994 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:40,003 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:40,005 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"d7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8","op":"PREPARE","stateRootHash":null,"ppTime":1503933459,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:40,044 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.491 seconds 2017-08-28 15:17:40,059 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:40,066 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,067 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933459, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,067 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,102 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 2) from Node4:1 2017-08-28 15:17:40,102 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 2) at 249756.251643411 2017-08-28 15:17:40,103 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:17:40,111 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,117 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:40,117 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,128 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:40,140 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,140 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933459, 'op': 'PREPARE', 'ppSeqNo': 2, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0}, 'Node2') 2017-08-28 15:17:40,144 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,150 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,156 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,157 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,160 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,160 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933459, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,173 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,185 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:40,203 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 2) from Node2:0 2017-08-28 15:17:40,210 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,210 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,210 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 2) from Node4:0 2017-08-28 15:17:40,222 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 2) at 249756.371090026 2017-08-28 15:17:40,222 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:17:40,229 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,229 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:40,229 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,241 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 2) from Node3:1 2017-08-28 15:17:40,242 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,242 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,253 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:40,260 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,269 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:17:40,273 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,295 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node2 2017-08-28 15:17:40,302 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node2 2017-08-28 15:17:40,303 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,306 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,310 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,311 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,326 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:40,327 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 2) from Node3:0 2017-08-28 15:17:40,327 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'd7bf713f6659981a5bb834262d99d6b4c145a719c34936a45c2f89f39a8f9ff8', 'viewNo': 0, 'instId': 0, 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ppTime': 1503933459, 'ppSeqNo': 2} 2017-08-28 15:17:40,339 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 2) 2017-08-28 15:17:40,341 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:40,350 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,358 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:17:40,359 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,365 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,369 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,371 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,372 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,372 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:40,383 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:40,399 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node2 2017-08-28 15:17:40,407 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node2 2017-08-28 15:17:40,407 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,408 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-08-28 15:17:40,422 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,422 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-08-28 15:17:40,428 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 2) from Node2:1 2017-08-28 15:17:40,438 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:40,439 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,442 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 2) from Node4:1 2017-08-28 15:17:40,443 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:17:40,453 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 2) 2017-08-28 15:17:40,454 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 2) 2017-08-28 15:17:40,454 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:17:40,461 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 2, 'ppTime': 1503933459, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:40,462 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 2) 2017-08-28 15:17:40,462 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,473 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:17:40,475 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:40,487 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,487 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:17:40,493 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,493 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,504 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:40,505 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,506 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,511 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:17:40,519 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:40,545 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:17:40,545 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 2) from Node2:0 2017-08-28 15:17:40,547 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:40,560 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,560 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 2) from Node3:0 2017-08-28 15:17:40,567 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:17:40,568 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 2) 2017-08-28 15:17:40,568 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 2) 2017-08-28 15:17:40,569 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:17:40,575 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 2, 'ppTime': 1503933459, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488)], 'stateRootHash': '6xyQiaB6vJ2TpyeNbqxvoGeLwnSC6YXqiMZK8oiXfSsR', 'txnRootHash': '32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:40,576 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 2) 2017-08-28 15:17:40,576 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 2) 2017-08-28 15:17:40,583 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 2, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:17:40,618 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 2 of 1 requests 2017-08-28 15:17:40,687 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:40,708 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) to client 2017-08-28 15:17:40,709 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"33DEkGqHn1PQVocRBgvEz91rSXKgw8GV68L71GJSn4NGKCnXy2UdP3amXysxv8UAhuxTG5iuAUzKcqtLjfa4CWXe","txnTime":1503933459,"auditPath":["A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","verkey":"~RmCt3RtDvKDfQBSKxo4qvy","seqNo":11,"type":"1","role":"2","rootHash":"32FqmAN6rDrGDKcGSmxrzCYjYjYWwU5jdxUJsjcpS3Bb","reqId":1503933459424488,"dest":"XhYtvJqezMUKfF6KVNaGmT"},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:40,715 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 2) for ledger 1 range (11, 11) 2017-08-28 15:17:40,731 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:40,813 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.758 seconds 2017-08-28 15:17:42,723 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:42,733 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,734 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:42,752 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:42,752 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:43,885 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 42 2017-08-28 15:17:43,888 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:43,899 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:43,900 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.7970697596790002 is acceptable. 2017-08-28 15:17:43,900 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:43,904 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:43,904 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:43,907 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 43 to run in 10 seconds 2017-08-28 15:17:46,208 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:17:46,217 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933466149129 2017-08-28 15:17:46,218 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,218 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,230 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,230 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,231 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933466149129 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:46,240 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:46,240 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since not finalised 2017-08-28 15:17:46,241 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:46,251 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node2 2017-08-28 15:17:46,264 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node2 2017-08-28 15:17:46,264 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node3 2017-08-28 15:17:46,264 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node3 2017-08-28 15:17:46,275 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node4 2017-08-28 15:17:46,276 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","operation":{"type":"1","role":"2","dest":"FBQ5W7LxRp8bLeVNEafjpd","verkey":"~2a4KepmnVVAPGo5gNA3P3p"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933466149129}}' to Node4 2017-08-28 15:17:46,278 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:46,293 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,303 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,303 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}}, 'Node2') 2017-08-28 15:17:46,304 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,306 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,315 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,320 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}}, 'Node4') 2017-08-28 15:17:46,321 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,322 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933466149129 2017-08-28 15:17:46,330 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,330 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}}, 'Node3') 2017-08-28 15:17:46,330 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,331 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129}} 2017-08-28 15:17:46,342 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,342 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to 0 replicas 2017-08-28 15:17:46,351 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,357 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,357 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since already forwarded 2017-08-28 15:17:46,357 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'reqId': 1503933466149129}} 2017-08-28 15:17:46,367 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} 2017-08-28 15:17:46,368 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi', 'operation': {'type': '1', 'role': '2', 'dest': 'FBQ5W7LxRp8bLeVNEafjpd', 'verkey': '~2a4KepmnVVAPGo5gNA3P3p'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933466149129} to its replicas since already forwarded 2017-08-28 15:17:46,370 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 3 for ledger 1 with state root b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`' 2017-08-28 15:17:46,390 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:46,404 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:17:46,404 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 1} 2017-08-28 15:17:46,404 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 1} with state root b'X\xa1)/\xfagp\x96\x03\xed_\xce\xf9\xe9\x80\xbb&M\x02Q\xcc\xc5&\xc7JD\x1fz5\x95\xa7`' 2017-08-28 15:17:46,433 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933466, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:46,467 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:46,468 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:46,468 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:46,469 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:46,473 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:46,478 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933466149129]],"stateRootHash":"HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4","ledgerId":1,"ppTime":1503933466,"ppSeqNo":3,"instId":0,"txnRootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:46,492 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:46,493 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 1} 2017-08-28 15:17:46,501 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'ppTime': 1503933466, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:17:46,501 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933466149129]], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 1} 2017-08-28 15:17:46,524 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 3) from Node2:1 at 249762.673425071 2017-08-28 15:17:46,525 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 3) at 249762.674102171 2017-08-28 15:17:46,533 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:17:46,533 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,533 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,543 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933466, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 1} with state root None 2017-08-28 15:17:46,544 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 3) 2017-08-28 15:17:46,547 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:46,587 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:46,588 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:46,594 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:46,594 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:46,595 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:46,595 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314","op":"PREPARE","stateRootHash":null,"ppTime":1503933466,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:46,606 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:46,618 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,625 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933466, 'op': 'PREPARE', 'ppSeqNo': 3, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0}, 'Node2') 2017-08-28 15:17:46,626 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,626 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,635 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933466, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,636 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,665 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 3) from Node2:0 2017-08-28 15:17:46,667 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,668 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,668 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 3) from Node4:1 2017-08-28 15:17:46,680 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 3) at 249762.828726427 2017-08-28 15:17:46,683 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:17:46,683 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:46,684 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:46,684 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,693 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:46,701 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,709 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933466, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:46,709 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,727 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,728 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:46,728 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,742 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:46,759 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 3) from Node4:0 2017-08-28 15:17:46,763 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 3) at 249762.911979764 2017-08-28 15:17:46,767 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:17:46,768 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:46,768 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:46,768 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,776 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 3) from Node3:1 2017-08-28 15:17:46,776 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,784 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,832 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node2 2017-08-28 15:17:46,833 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node2 2017-08-28 15:17:46,839 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,840 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,842 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,848 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,865 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:46,866 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:46,876 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:46,881 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:17:46,881 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:46,893 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,894 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:17:46,894 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,922 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node2 2017-08-28 15:17:46,930 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node2 2017-08-28 15:17:46,932 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,933 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-08-28 15:17:46,946 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,947 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-08-28 15:17:46,964 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 3) from Node3:0 2017-08-28 15:17:46,965 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '6d51061c2a2b0ce99b783e67d2c859ed53861d7fb9304bd6407b1e8088ff5314', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ppTime': 1503933466, 'ppSeqNo': 3} 2017-08-28 15:17:46,971 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 3) 2017-08-28 15:17:46,972 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 3) from Node2:1 2017-08-28 15:17:46,972 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:46,985 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:46,986 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:46,993 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:47,006 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:17:47,006 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:47,006 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,007 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:47,016 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,016 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:47,017 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:17:47,026 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:47,031 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,032 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:17:47,032 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,041 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:17:47,041 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:17:47,050 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:17:47,050 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,051 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:47,055 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:47,056 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:47,056 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:47,067 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:47,089 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 3) from Node4:0 2017-08-28 15:17:47,099 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:47,100 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,100 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 3) from Node2:0 2017-08-28 15:17:47,111 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:17:47,112 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 3) 2017-08-28 15:17:47,119 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 3) 2017-08-28 15:17:47,120 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:17:47,124 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 3, 'ppTime': 1503933466, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'stateRootHash': 'HBVJC32863gACy84Y9ytWz7uhpfCXcp9jCB3ZB9VU1d4', 'txnRootHash': '8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:47,124 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 3) 2017-08-28 15:17:47,124 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,125 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:17:47,128 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 3) from Node4:1 2017-08-28 15:17:47,128 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:17:47,129 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 3) 2017-08-28 15:17:47,129 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 3) 2017-08-28 15:17:47,131 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:17:47,132 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 3, 'ppTime': 1503933466, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:47,132 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 3) 2017-08-28 15:17:47,132 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 3) 2017-08-28 15:17:47,156 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 3, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:17:47,220 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 3 of 1 requests 2017-08-28 15:17:47,240 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:47,248 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) to client 2017-08-28 15:17:47,248 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"dXFex68iuW2sNXJPJaAq1zxyYfxk7s4ss89ydcmfe2NS7L5RndxCJhc6VGvRWH6cEArg5SowGE4JuPLCD87fgQi","txnTime":1503933466,"auditPath":["8Bzurgoq64vp4oTqg9QmtZkGLuK4QzErdQJfLftvozvY","A7qtYTZtZNYspt397HsDzan5USLKgejsseghD1sboLRw","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","verkey":"~2a4KepmnVVAPGo5gNA3P3p","seqNo":12,"type":"1","role":"2","rootHash":"8sQXSyumGNAaWvATVR2fFDhEQmbKAmF9qtCDBRp7Utjg","reqId":1503933466149129,"dest":"FBQ5W7LxRp8bLeVNEafjpd"},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:47,263 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 3) for ledger 1 range (12, 12) 2017-08-28 15:17:47,277 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:17:47,280 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:47,330 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.149 seconds 2017-08-28 15:17:51,727 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:17:51,739 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933471684615 2017-08-28 15:17:51,742 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,743 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,747 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,748 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,750 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933471684615 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:51,760 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:51,765 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since not finalised 2017-08-28 15:17:51,766 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:51,787 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node2 2017-08-28 15:17:51,788 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node2 2017-08-28 15:17:51,795 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node3 2017-08-28 15:17:51,796 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node3 2017-08-28 15:17:51,796 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node4 2017-08-28 15:17:51,797 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","operation":{"type":"1","role":"100","dest":"6XiqM9DwYDgFnmyQQBFr72","verkey":"~YHDYsnaN59TzUsLqvPYdRr"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933471684615}}' to Node4 2017-08-28 15:17:51,807 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:51,826 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,827 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,830 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}}, 'Node2') 2017-08-28 15:17:51,833 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,845 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,845 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,849 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}}, 'Node4') 2017-08-28 15:17:51,849 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,852 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933471684615 2017-08-28 15:17:51,861 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,861 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}}, 'Node3') 2017-08-28 15:17:51,861 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,869 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615}} 2017-08-28 15:17:51,876 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,877 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to 0 replicas 2017-08-28 15:17:51,886 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,887 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,891 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since already forwarded 2017-08-28 15:17:51,892 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'reqId': 1503933471684615}} 2017-08-28 15:17:51,901 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} 2017-08-28 15:17:51,902 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku', 'operation': {'type': '1', 'role': '100', 'dest': '6XiqM9DwYDgFnmyQQBFr72', 'verkey': '~YHDYsnaN59TzUsLqvPYdRr'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933471684615} to its replicas since already forwarded 2017-08-28 15:17:51,917 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 4 for ledger 1 with state root b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3' 2017-08-28 15:17:51,954 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:51,955 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:17:51,955 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'instId': 0, 'ppSeqNo': 4, 'ledgerId': 1} 2017-08-28 15:17:51,957 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'instId': 0, 'ppSeqNo': 4, 'ledgerId': 1} with state root b'\xf0j\xcb2?0z\xd5\xd8\x18\xbb\xcfD\x81zF\xd5\x1a\x82~Z\xf72p\xe4\xa9\x05\x85\xf6\x0c\x15\xa3' 2017-08-28 15:17:51,972 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933471, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'instId': 0, 'ppSeqNo': 4, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:51,987 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:52,001 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 4, 'ledgerId': 1} 2017-08-28 15:17:52,002 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 4, 'ppTime': 1503933471, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:17:52,002 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933471684615]], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 4, 'ledgerId': 1} 2017-08-28 15:17:52,021 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:52,022 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:52,022 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:52,031 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:52,032 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:52,032 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933471684615]],"stateRootHash":"HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV","ledgerId":1,"ppTime":1503933471,"ppSeqNo":4,"instId":0,"txnRootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:52,044 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 4) from Node2:1 at 249768.193545281 2017-08-28 15:17:52,052 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 4) at 249768.200687602 2017-08-28 15:17:52,052 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:17:52,059 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,060 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,060 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933471, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 4, 'ledgerId': 1} with state root None 2017-08-28 15:17:52,071 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 4) 2017-08-28 15:17:52,075 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:52,115 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:52,119 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:52,127 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:52,128 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:52,128 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:52,133 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a","op":"PREPARE","stateRootHash":null,"ppTime":1503933471,"ppSeqNo":4,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:52,155 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.462 seconds 2017-08-28 15:17:52,167 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:52,182 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,182 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,189 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,209 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 4) from Node3:1 2017-08-28 15:17:52,210 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 4) at 249768.358975101 2017-08-28 15:17:52,219 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:17:52,220 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,220 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:52,221 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,235 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:52,239 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,240 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933471, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,253 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,253 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,254 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4}, 'Node3') 2017-08-28 15:17:52,254 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,277 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:52,290 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 4) from Node3:0 2017-08-28 15:17:52,290 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,301 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,302 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 4) from Node4:1 2017-08-28 15:17:52,302 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,303 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,319 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:52,320 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,327 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933471, 'op': 'PREPARE', 'ppSeqNo': 4, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0}, 'Node2') 2017-08-28 15:17:52,328 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,328 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,335 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:17:52,341 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,341 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933471, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,342 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,365 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node2 2017-08-28 15:17:52,366 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node2 2017-08-28 15:17:52,366 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,379 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,380 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,380 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,389 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:17:52,390 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:17:52,405 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 4) from Node2:0 2017-08-28 15:17:52,405 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 4) at 249768.554297868 2017-08-28 15:17:52,411 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:17:52,412 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,412 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:52,413 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,423 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 4) from Node4:0 2017-08-28 15:17:52,424 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '7e203863d757792723a9df1a10affd03aa368e211d02b8a9e60eafb3b0cdba9a', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ppTime': 1503933471, 'ppSeqNo': 4} 2017-08-28 15:17:52,432 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 4) 2017-08-28 15:17:52,434 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:52,446 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,460 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,460 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,471 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,472 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:52,472 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,509 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:52,512 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 4) from Node4:1 2017-08-28 15:17:52,519 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:52,520 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,520 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 4) from Node3:1 2017-08-28 15:17:52,523 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:17:52,527 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 4) 2017-08-28 15:17:52,531 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 4) 2017-08-28 15:17:52,532 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:17:52,532 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 4, 'ppTime': 1503933471, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:52,543 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 4) 2017-08-28 15:17:52,544 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,545 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:52,562 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,563 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:17:52,576 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:52,576 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,577 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 4}, 'Node4') 2017-08-28 15:17:52,582 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,583 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,591 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:52,592 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,619 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node2 2017-08-28 15:17:52,621 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node2 2017-08-28 15:17:52,621 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,622 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node3 2017-08-28 15:17:52,632 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,632 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":4}' to Node4 2017-08-28 15:17:52,641 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:17:52,653 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 4) from Node4:0 2017-08-28 15:17:52,654 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:52,654 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,655 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 4) from Node3:0 2017-08-28 15:17:52,655 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:17:52,668 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 4) 2017-08-28 15:17:52,668 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 4) 2017-08-28 15:17:52,668 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:17:52,669 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 4, 'ppTime': 1503933471, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615)], 'stateRootHash': 'HZ3KiJYfhAMe8FszHXNHh1kyFCKKnGoSsv1WsysHmFWV', 'txnRootHash': 'CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:52,675 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 4) 2017-08-28 15:17:52,677 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 4) 2017-08-28 15:17:52,687 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:17:52,688 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:52,707 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,707 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:17:52,707 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:52,714 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 4 of 1 requests 2017-08-28 15:17:52,715 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:52,740 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) to client 2017-08-28 15:17:52,740 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"4nubSjtZvYdVxRdaGWGMLKruPpWZMJ9JTJK94QstXMKMt3Fe13qmUNzzKacSD1VZVk7L89caYvYjPdwfAYXsf1ku","txnTime":1503933471,"auditPath":["4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","verkey":"~YHDYsnaN59TzUsLqvPYdRr","seqNo":13,"type":"1","role":"100","rootHash":"CcZdAMCj7rZjuyVUAgtF3KM7K1VhZP4hFb5HzCQVH5XA","reqId":1503933471684615,"dest":"6XiqM9DwYDgFnmyQQBFr72"},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:52,757 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 4) for ledger 1 range (13, 13) 2017-08-28 15:17:52,757 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:52,758 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 4, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:17:52,805 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.639 seconds 2017-08-28 15:17:53,244 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 37 2017-08-28 15:17:53,244 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:17:53,245 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:53,248 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 44 to run in 60 seconds 2017-08-28 15:17:53,249 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 6 2017-08-28 15:17:53,263 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:17:53,264 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-08-28 15:17:53,723 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:17:53,734 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:17:53,734 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:17:53,735 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:17:53,930 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 43 2017-08-28 15:17:53,931 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:17:53,932 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:17:53,935 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.8752667561678897 is acceptable. 2017-08-28 15:17:53,935 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:53,937 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:53,937 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:17:53,945 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 45 to run in 10 seconds 2017-08-28 15:17:57,603 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:57,614 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil': INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:17:57,621 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil') 2017-08-28 15:17:57,621 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:17:57,633 | DEBUG | node.py (1896) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:57,634 | DEBUG | node.py (1939) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-08-28 15:17:57,634 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.8752667561678897 is acceptable. 2017-08-28 15:17:57,634 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:17:57,635 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:17:57,635 | DEBUG | node.py (1927) | processInstanceChange | Node1 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:17:57,837 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:57,843 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:57,845 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:17:57,864 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:17:57,865 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:17:58,906 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:17:58,910 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933478716086 2017-08-28 15:17:58,911 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,930 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,931 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,932 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:58,935 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933478716086 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:17:58,945 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:58,946 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since not finalised 2017-08-28 15:17:58,957 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:58,963 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node2 2017-08-28 15:17:58,968 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node2 2017-08-28 15:17:58,972 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node3 2017-08-28 15:17:58,972 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node3 2017-08-28 15:17:58,983 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node4 2017-08-28 15:17:58,989 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","operation":{"type":"1","role":"101","dest":"Rhx2qwaeiGyhU9vn4ynHSS","verkey":"~RTJS62PRHurqHJnbnT1Hgi"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933478716086}}' to Node4 2017-08-28 15:17:59,004 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:59,005 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,018 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,023 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}}, 'Node4') 2017-08-28 15:17:59,025 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,026 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,027 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,027 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to 0 replicas 2017-08-28 15:17:59,046 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 5 for ledger 1 with state root b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6' 2017-08-28 15:17:59,063 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:17:59,075 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:17:59,076 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'instId': 0, 'ppSeqNo': 5, 'ledgerId': 1} 2017-08-28 15:17:59,078 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'instId': 0, 'ppSeqNo': 5, 'ledgerId': 1} with state root b'\xf5\xf03\xf1\xe5\xcf\xf4gk\x0evA\x08\xc2;o\x06o\xcequ\xf5R:\xbf\x86\xea\x9f\x1f#C\xc6' 2017-08-28 15:17:59,084 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:59,086 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,099 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,100 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}}, 'Node3') 2017-08-28 15:17:59,100 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,101 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933478716086 2017-08-28 15:17:59,116 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,116 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}}, 'Node2') 2017-08-28 15:17:59,116 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,117 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'reqId': 1503933478716086}} 2017-08-28 15:17:59,120 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,134 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since already forwarded 2017-08-28 15:17:59,134 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086}} 2017-08-28 15:17:59,135 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} 2017-08-28 15:17:59,146 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW', 'operation': {'type': '1', 'role': '101', 'dest': 'Rhx2qwaeiGyhU9vn4ynHSS', 'verkey': '~RTJS62PRHurqHJnbnT1Hgi'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933478716086} to its replicas since already forwarded 2017-08-28 15:17:59,152 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933479, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'instId': 0, 'ppSeqNo': 5, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:59,185 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:59,186 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:17:59,188 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:59,188 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:17:59,194 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:59,194 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933478716086]],"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1","ledgerId":1,"ppTime":1503933479,"ppSeqNo":5,"instId":0,"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:17:59,214 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:17:59,215 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 5, 'ledgerId': 1} 2017-08-28 15:17:59,215 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 5, 'ppTime': 1503933479, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:17:59,223 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933478716086]], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 5, 'ledgerId': 1} 2017-08-28 15:17:59,247 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 5) from Node2:1 at 249775.396538974 2017-08-28 15:17:59,248 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 5) at 249775.397314688 2017-08-28 15:17:59,248 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:17:59,259 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,260 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,260 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933479, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 5, 'ledgerId': 1} with state root None 2017-08-28 15:17:59,267 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 5) 2017-08-28 15:17:59,281 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:59,317 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:59,324 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:17:59,331 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:59,332 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:17:59,340 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:59,340 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","op":"PREPARE","stateRootHash":null,"ppTime":1503933479,"ppSeqNo":5,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:17:59,377 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.495 seconds 2017-08-28 15:17:59,389 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:59,400 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,401 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933479, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,409 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,410 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,410 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933479, 'op': 'PREPARE', 'ppSeqNo': 5, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0}, 'Node2') 2017-08-28 15:17:59,424 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,442 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 5) from Node2:0 2017-08-28 15:17:59,460 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,460 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,462 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 5) from Node4:1 2017-08-28 15:17:59,476 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 5) at 249775.625037365 2017-08-28 15:17:59,476 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:17:59,477 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,477 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:59,481 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,482 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:59,495 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,496 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933479, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,503 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,503 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']} 2017-08-28 15:17:59,509 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:17:59,515 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":0,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":"96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":5,"instId":1,"digest":"298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210","ppTime":1503933479,"stateRootHash":null}']} 2017-08-28 15:17:59,523 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,524 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,532 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,532 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,532 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5}, 'Node3') 2017-08-28 15:17:59,547 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,550 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:59,564 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 5) from Node4:0 2017-08-28 15:17:59,570 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 5) at 249775.718678889 2017-08-28 15:17:59,570 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:17:59,570 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,577 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:59,583 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,583 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 5) from Node3:0 2017-08-28 15:17:59,591 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 0, 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,592 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,598 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 5) from Node3:1 2017-08-28 15:17:59,599 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '298b71566e853d0ec86db791dcb54f49aec04e0a554aa61f0af2d7641c50b210', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933479, 'ppSeqNo': 5} 2017-08-28 15:17:59,605 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 5) 2017-08-28 15:17:59,620 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node2 2017-08-28 15:17:59,621 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node2 2017-08-28 15:17:59,632 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,632 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,643 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,644 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,652 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:17:59,662 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:17:59,674 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,675 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:17:59,683 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,684 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:59,684 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:17:59,684 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":5,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":5,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:17:59,685 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,685 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,697 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,697 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,698 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 5}, 'Node4') 2017-08-28 15:17:59,708 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,709 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,718 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 5, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:59,718 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} 2017-08-28 15:17:59,739 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node2 2017-08-28 15:17:59,740 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node2 2017-08-28 15:17:59,746 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,747 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node3 2017-08-28 15:17:59,747 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":5}' to Node4 2017-08-28 15:17:59,759 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 5) from Node4:0 2017-08-28 15:17:59,773 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:17:59,774 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,780 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 5) from Node2:1 2017-08-28 15:17:59,780 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:17:59,780 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,788 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 5) from Node4:1 2017-08-28 15:17:59,788 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:17:59,788 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 5) 2017-08-28 15:17:59,793 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 5) 2017-08-28 15:17:59,794 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:17:59,812 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 5, 'ppTime': 1503933479, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:59,812 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 5) 2017-08-28 15:17:59,812 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,819 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:17:59,824 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:17:59,826 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,837 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 5, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:17:59,838 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,838 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,842 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 5, 'viewNo': 0}, 'Node3') 2017-08-28 15:17:59,852 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} 2017-08-28 15:17:59,873 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:17:59,883 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 5) from Node2:0 2017-08-28 15:17:59,884 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:17:59,884 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 5) 2017-08-28 15:17:59,899 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 5) 2017-08-28 15:17:59,902 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:17:59,902 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 5, 'ppTime': 1503933479, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086)], 'stateRootHash': '96wAvETh4TF256sck2E24C1jz9qJuuu1t3ffha3XMxv1', 'txnRootHash': 'Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:17:59,902 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 5) 2017-08-28 15:17:59,906 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 5) 2017-08-28 15:17:59,906 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 5, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:17:59,958 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 5 of 1 requests 2017-08-28 15:17:59,960 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:17:59,964 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) to client 2017-08-28 15:17:59,964 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"5QCYSHuA6UfU5wYZ86osbmhHADHiXfD5mUqjR7syPBnbqhJzJuzrzHtL3L3VeeoiTR4ybMJGM6RYnPG2JKjtSpmW","txnTime":1503933479,"auditPath":["HrVYnCQ4sRNzhQigrJsjEPBmrAWsPa9H1xsgfef3LEMw","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","verkey":"~RTJS62PRHurqHJnbnT1Hgi","seqNo":14,"type":"1","role":"101","rootHash":"Ey76jy7phAp1GUMfoaRt7wAsowWeMVjTmJTgSkGdJPXn","reqId":1503933478716086,"dest":"Rhx2qwaeiGyhU9vn4ynHSS"},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:17:59,986 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 5) for ledger 1 range (14, 14) 2017-08-28 15:17:59,987 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:00,035 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.647 seconds 2017-08-28 15:18:03,973 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 45 2017-08-28 15:18:03,973 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:03,974 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:03,975 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node1 master throughput ratio 0.8787177431304901 is acceptable. 2017-08-28 15:18:03,975 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:03,976 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:03,994 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:03,995 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-08-28 15:18:04,213 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:18:04,235 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933484172953 2017-08-28 15:18:04,236 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,237 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,247 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,248 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,258 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933484172953 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:04,258 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:04,266 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since not finalised 2017-08-28 15:18:04,267 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:04,278 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node2 2017-08-28 15:18:04,278 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node2 2017-08-28 15:18:04,284 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node3 2017-08-28 15:18:04,291 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node3 2017-08-28 15:18:04,291 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node4 2017-08-28 15:18:04,300 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","operation":{"type":"1","dest":"5qhSKMkpqui6taGLTecwqf","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933484172953}}' to Node4 2017-08-28 15:18:04,317 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:04,329 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,330 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,330 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}}, 'Node4') 2017-08-28 15:18:04,330 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,339 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,347 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,348 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}}, 'Node2') 2017-08-28 15:18:04,348 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,349 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933484172953 2017-08-28 15:18:04,357 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,357 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}}, 'Node3') 2017-08-28 15:18:04,359 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,360 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,369 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,370 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to 0 replicas 2017-08-28 15:18:04,375 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953}} 2017-08-28 15:18:04,381 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,381 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since already forwarded 2017-08-28 15:18:04,382 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'reqId': 1503933484172953}} 2017-08-28 15:18:04,382 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} 2017-08-28 15:18:04,382 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x', 'operation': {'type': '1', 'dest': '5qhSKMkpqui6taGLTecwqf', 'verkey': '~UWK9xwiKsqMQSBmymYpR5v'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933484172953} to its replicas since already forwarded 2017-08-28 15:18:04,405 | DEBUG | replica.py ( 665) | create3PCBatch | Node1:0 creating batch 6 for ledger 1 with state root b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>' 2017-08-28 15:18:04,431 | DEBUG | replica.py ( 698) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-08-28 15:18:04,432 | DEBUG | replica.py (2133) | send | Node1:0 sending PrePrepare 2017-08-28 15:18:04,432 | TRACE | replica.py (2134) | send | Node1:0 sending PREPREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'instId': 0, 'ppSeqNo': 6, 'ledgerId': 1} 2017-08-28 15:18:04,432 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'instId': 0, 'ppSeqNo': 6, 'ledgerId': 1} with state root b'xb\xa6\x9a\xa4{\xde\xdcZ\xa7\xa2|\x86\x9b<\xff\x11\xe6\xba\xa5\xc5\xf3\x17|!\xac\x10s\xcc\xb5q>' 2017-08-28 15:18:04,438 | DEBUG | node.py (2566) | send | Node1 sending message PREPREPARE{'ppTime': 1503933484, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'instId': 0, 'ppSeqNo': 6, 'ledgerId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:04,483 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:18:04,488 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node2 2017-08-28 15:18:04,488 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:18:04,500 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node3 2017-08-28 15:18:04,501 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:18:04,501 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPREPARE","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1503933484172953]],"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz","ledgerId":1,"ppTime":1503933484,"ppSeqNo":6,"instId":0,"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","discarded":1,"viewNo":0}' to Node4 2017-08-28 15:18:04,523 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:04,524 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 6, 'ledgerId': 1} 2017-08-28 15:18:04,529 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 6, 'ppTime': 1503933484, 'op': 'PREPREPARE', 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'stateRootHash': None, 'txnRootHash': None, 'discarded': 1, 'viewNo': 0, 'ledgerId': 1}, 'Node2') 2017-08-28 15:18:04,529 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933484172953]], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 6, 'ledgerId': 1} 2017-08-28 15:18:04,552 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(0, 6) from Node2:1 at 249780.701206013 2017-08-28 15:18:04,553 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(0, 6) at 249780.702218586 2017-08-28 15:18:04,566 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:18:04,566 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,566 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,567 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933484, 'stateRootHash': None, 'viewNo': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 6, 'ledgerId': 1} with state root None 2017-08-28 15:18:04,567 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(0, 6) 2017-08-28 15:18:04,594 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:04,614 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:18:04,621 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node2 2017-08-28 15:18:04,621 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:18:04,625 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node3 2017-08-28 15:18:04,626 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:18:04,626 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","op":"PREPARE","stateRootHash":null,"ppTime":1503933484,"ppSeqNo":6,"instId":1,"txnRootHash":null,"viewNo":0}' to Node4 2017-08-28 15:18:04,646 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:04,647 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,659 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933484, 'op': 'PREPARE', 'ppSeqNo': 6, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0}, 'Node2') 2017-08-28 15:18:04,659 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,681 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 6) from Node2:0 2017-08-28 15:18:04,682 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,682 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,692 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:04,696 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,703 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933484, 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,708 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,732 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 6) from Node4:1 2017-08-28 15:18:04,732 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(0, 6) at 249780.881321974 2017-08-28 15:18:04,732 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:18:04,732 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:04,733 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:18:04,733 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,745 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:04,753 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']} 2017-08-28 15:18:04,760 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:04,761 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"txnRootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":0,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":"FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz"}', '{"txnRootHash":null,"viewNo":0,"op":"PREPARE","ppSeqNo":6,"instId":1,"digest":"456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337","ppTime":1503933484,"stateRootHash":null}']} 2017-08-28 15:18:04,776 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,779 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,780 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,780 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,791 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'op': 'PREPARE', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6}, 'Node3') 2017-08-28 15:18:04,795 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,816 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:04,823 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 6) from Node3:0 2017-08-28 15:18:04,828 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 6) at 249780.976879163 2017-08-28 15:18:04,828 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:18:04,828 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:04,831 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:18:04,840 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,840 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(0, 6) from Node3:1 2017-08-28 15:18:04,842 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,852 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,853 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:04,864 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,874 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933484, 'op': 'PREPARE', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,874 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,900 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node2 2017-08-28 15:18:04,900 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node2 2017-08-28 15:18:04,907 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:04,912 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:04,917 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:04,917 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:04,919 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:04,929 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(0, 6) from Node4:0 2017-08-28 15:18:04,929 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '456ff2450fe743c4ba18654dad6446038f0cc9b9ffa68096c22db425121d9337', 'viewNo': 0, 'instId': 0, 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ppTime': 1503933484, 'ppSeqNo': 6} 2017-08-28 15:18:04,935 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 6) 2017-08-28 15:18:04,950 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:04,951 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:04,955 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'viewNo': 0, 'instId': 1}, 'Node2') 2017-08-28 15:18:04,956 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:04,956 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:18:04,956 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:04,956 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":6,"instId":0,"viewNo":0,"op":"COMMIT"}', '{"ppSeqNo":6,"instId":1,"viewNo":0,"op":"COMMIT"}']} 2017-08-28 15:18:04,957 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:04,957 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 0, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,957 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:04,972 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:04,973 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 0, 'instId': 1, 'ppSeqNo': 6}, 'Node4') 2017-08-28 15:18:04,973 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:04,986 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']} 2017-08-28 15:18:04,986 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:04,991 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":0}', '{"op":"COMMIT","viewNo":0,"ppSeqNo":6,"instId":1}']} 2017-08-28 15:18:04,992 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:04,992 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 6, 'viewNo': 0}, 'Node3') 2017-08-28 15:18:04,992 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:05,002 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:05,002 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 6, 'viewNo': 0}, 'Node3') 2017-08-28 15:18:05,011 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} 2017-08-28 15:18:05,035 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node2 2017-08-28 15:18:05,036 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node2 2017-08-28 15:18:05,048 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:05,048 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node3 2017-08-28 15:18:05,057 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:05,058 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":6}' to Node4 2017-08-28 15:18:05,069 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 6) from Node4:0 2017-08-28 15:18:05,070 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} commits where f is 1 2017-08-28 15:18:05,079 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,083 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 6) from Node3:0 2017-08-28 15:18:05,084 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:18:05,084 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (0, 6) 2017-08-28 15:18:05,093 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (0, 6) 2017-08-28 15:18:05,094 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:18:05,094 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 6, 'ppTime': 1503933484, 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'stateRootHash': 'FyhHLtJxiMMjFPLzkXtnuEoze3xJmTBLoygPj92a2YMz', 'txnRootHash': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:18:05,101 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (0, 6) 2017-08-28 15:18:05,102 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,102 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 6) from Node2:1 2017-08-28 15:18:05,103 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} commits where f is 1 2017-08-28 15:18:05,111 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,112 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(0, 6) from Node4:1 2017-08-28 15:18:05,112 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:18:05,124 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (0, 6) 2017-08-28 15:18:05,124 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (0, 6) 2017-08-28 15:18:05,135 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:18:05,136 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 6, 'ppTime': 1503933484, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0} 2017-08-28 15:18:05,136 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (0, 6) 2017-08-28 15:18:05,136 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(0, 6) 2017-08-28 15:18:05,139 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:18:05,141 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:05,149 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:05,149 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 6, 'op': 'COMMIT', 'viewNo': 0, 'instId': 0}, 'Node2') 2017-08-28 15:18:05,162 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} 2017-08-28 15:18:05,177 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 0 6 of 1 requests 2017-08-28 15:18:05,189 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-08-28 15:18:05,189 | DEBUG | node.py (2344) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) to client 2017-08-28 15:18:05,200 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"result":{"signature":"4bFhS6WMQ2uMTKhy5Np98naJNhtiQXFWbizbUoDKML7QK8HZs9RTSnu7uEVCSovQfEzNtT1QCbuu3BoQgY1hRW2x","txnTime":1503933484,"auditPath":["ijqq5QRv5GQwp9riPYGkR5Xkz9QxMds39Wc21R4V9Pt","4DUBvjuKvdvzPKForxX5ArBcFEbXEKk9NrMQy26SkxLK","FUUbzChmnGjrGChBv3LZoKunodBPrVuMcg2vUrhkndmz"],"identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"5qhSKMkpqui6taGLTecwqf","seqNo":15,"type":"1","reqId":1503933484172953,"rootHash":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","verkey":"~UWK9xwiKsqMQSBmymYpR5v"},"op":"REPLY"}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:05,224 | DEBUG | node.py (2263) | executeBatch | Node1 storing 3PC key (0, 6) for ledger 1 range (15, 15) 2017-08-28 15:18:05,224 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:18:05,224 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:05,225 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 6, 'viewNo': 0, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:18:05,283 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 1.065 seconds 2017-08-28 15:18:07,162 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:07,170 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7': INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:18:07,186 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7') 2017-08-28 15:18:07,191 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-08-28 15:18:07,200 | DEBUG | node.py (1896) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:07,200 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 1 from 0 2017-08-28 15:18:07,217 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action _check_view_change_completed with id 47 to run in 60 seconds 2017-08-28 15:18:07,217 | DEBUG | replica.py ( 453) | on_view_change_start | Node1:0 setting last prepared for master to (0, 6) 2017-08-28 15:18:07,219 | DEBUG | node.py (2127) | startViewChange | Node1 resetting monitor stats after view change 2017-08-28 15:18:07,219 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:18:07,222 | DEBUG | replica.py ( 406) | primaryName | Node1:0 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,223 | DEBUG | replica.py ( 406) | primaryName | Node1:1 setting primaryName for view no 1 to: None 2017-08-28 15:18:07,336 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node1 changed to view 1, will start catchup now 2017-08-28 15:18:07,336 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,336 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,336 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,336 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,336 | DEBUG | node.py (2173) | start_catchup | Node1 reverted 0 batches before starting catch up 2017-08-28 15:18:07,337 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:07,337 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 0 2017-08-28 15:18:07,337 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,337 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 0 2017-08-28 15:18:07,337 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,337 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 0 2017-08-28 15:18:07,337 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,338 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 0 2017-08-28 15:18:07,338 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:07,338 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:07,338 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:07,338 | DEBUG | node.py (1927) | processInstanceChange | Node1 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:18:07,367 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,368 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,368 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,368 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:07,369 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,378 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,379 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,387 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:07,388 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,395 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,399 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,400 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:07,410 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:07,417 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:07,417 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:07,418 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:07,431 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,435 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,436 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,436 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,436 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,444 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,445 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,454 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:07,454 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,455 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,455 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,459 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,466 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,466 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,466 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,474 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,474 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:07,475 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,486 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,486 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:18:07,486 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,487 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,487 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:18:07,489 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,498 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,498 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,511 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,524 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,535 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,540 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,542 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,562 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,567 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,568 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:07,571 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,579 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,584 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,593 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:07,594 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,607 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:18:07,609 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:18:07,619 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:07,634 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:07,647 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,652 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:07,652 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:18:07,658 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,659 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:18:07,666 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,667 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,667 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:18:07,677 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,678 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:07,678 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:07,688 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:07,689 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,689 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,699 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,699 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,705 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:18:07,706 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,706 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node2 2017-08-28 15:18:07,713 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,714 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,714 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node2 2017-08-28 15:18:07,724 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,725 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,738 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node4 2017-08-28 15:18:07,738 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,739 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,740 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:18:07,746 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-08-28 15:18:07,746 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:18:07,746 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,746 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:07,746 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:07,747 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:07,747 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:18:07,747 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:18:07,747 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:18:07,747 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,765 | DEBUG | node.py ( 923) | checkInstances | Node1 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node3', 'Node4'} 2017-08-28 15:18:07,784 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:07,785 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-08-28 15:18:07,785 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:18:07,785 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-08-28 15:18:07,787 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:07,788 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-08-28 15:18:07,788 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:07,788 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 2 2017-08-28 15:18:07,788 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:18:07,799 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:18:07,801 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node4 2017-08-28 15:18:07,802 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,802 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,813 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:07,825 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,826 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:07,829 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:07,833 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:07,839 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,840 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:07,840 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:07,847 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:07,852 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:07,852 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:07,859 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:07,866 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:07,883 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:18:07,884 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:07,884 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:18:07,891 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,892 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,892 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,903 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,907 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:18:07,907 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:18:07,908 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node3 2017-08-28 15:18:07,918 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,919 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,925 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:18:07,926 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node3 2017-08-28 15:18:07,926 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,935 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:18:07,936 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:18:07,961 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:07,962 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,972 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:07,973 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:07,974 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,986 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:07,986 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,987 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,991 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:07,997 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:07,998 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:07,998 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:07,999 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:07,999 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,001 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:18:08,014 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,014 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,014 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:18:08,019 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,020 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,020 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:08,020 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,022 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,031 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,032 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,038 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,039 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,042 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,047 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,052 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,052 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,071 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,072 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,081 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,089 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,106 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,107 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:08,113 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:08,120 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,120 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,134 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:08,137 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:08,137 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,138 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:18:08,138 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:18:08,148 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:08,175 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:08,180 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:08,183 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:08,188 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:18:08,188 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,189 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:18:08,189 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,189 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,189 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:18:08,189 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,190 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,190 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:08,190 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,190 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,190 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:08,190 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,191 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,191 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:18:08,227 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,228 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:18:08,238 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:08,243 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:18:08,244 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,245 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,253 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,254 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,261 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:18:08,262 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:18:08,273 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:18:08,274 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,287 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,290 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:18:08,292 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,301 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,301 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:18:08,304 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,313 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,313 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:18:08,314 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-08-28 15:18:08,322 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,322 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,322 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,323 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,331 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,332 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:18:08,332 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:18:08,333 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:18:08,346 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,346 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:18:08,347 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-08-28 15:18:08,347 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:08,357 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-08-28 15:18:08,358 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,358 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-08-28 15:18:08,358 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,358 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-08-28 15:18:08,358 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,359 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 1 2017-08-28 15:18:08,359 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:18:08,359 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-08-28 15:18:08,359 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:18:08,386 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:18:08,394 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,395 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,396 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:18:08,409 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,409 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,411 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:18:08,420 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:18:08,420 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,421 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:18:08,421 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:18:08,440 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,441 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,452 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,452 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:08,460 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,460 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,461 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,464 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:08,470 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,471 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,471 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,480 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:08,482 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:08,498 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,504 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:08,504 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,504 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,518 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:18:08,518 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,519 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,523 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:18:08,527 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,539 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,543 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:18:08,550 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:18:08,551 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,560 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,569 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:18:08,573 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:08,591 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:08,600 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:08,600 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:18:08,600 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,610 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,610 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,611 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,612 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,612 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,621 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,622 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:08,622 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:18:08,624 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,634 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,636 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,636 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,636 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,637 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,651 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,652 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:18:08,653 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,661 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:18:08,679 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:18:08,679 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,680 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,691 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:18:08,696 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:18:08,696 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:18:08,701 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:18:08,702 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:18:08,713 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:08,714 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,714 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:18:08,714 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:18:08,741 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,741 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:18:08,747 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,748 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,749 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:18:08,749 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,749 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:18:08,759 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:18:08,760 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:18:08,760 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,771 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,772 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,772 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,779 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:18:08,780 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,780 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,788 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:18:08,788 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:18:08,801 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,802 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,802 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,802 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,811 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:18:08,812 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:18:08,812 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node2 2017-08-28 15:18:08,824 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,824 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,831 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node2 2017-08-28 15:18:08,832 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,833 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,838 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node3 2017-08-28 15:18:08,839 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,851 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,854 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:18:08,854 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} that it does not need catchup 2017-08-28 15:18:08,855 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:18:08,860 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,860 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:18:08,860 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:18:08,861 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:18:08,861 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:18:08,871 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:18:08,871 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:18:08,872 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,872 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:18:08,882 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,882 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:18:08,883 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:18:08,883 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 3 ViewChangeDone messages 2017-08-28 15:18:08,889 | DEBUG | node.py (1550) | caught_up_for_current_view | Node1 does not have view change quorum for view 1 2017-08-28 15:18:08,889 | DEBUG | node.py (1534) | is_catchup_needed | Node1 is not caught up for the current view 1 2017-08-28 15:18:08,890 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:18:08,890 | DEBUG | node.py (1539) | is_catchup_needed | Node1 ordered till last prepared certificate 2017-08-28 15:18:08,904 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-08-28 15:18:08,904 | DEBUG | primary_decider.py ( 131) | send | Node1's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:08,907 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 2 ViewChangeDone messages 2017-08-28 15:18:08,912 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:08,913 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node3 2017-08-28 15:18:08,920 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,920 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,928 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node4 2017-08-28 15:18:08,929 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,933 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,933 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:18:08,934 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node4 2017-08-28 15:18:08,946 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,946 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:18:08,947 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:18:08,962 | DEBUG | node.py (2566) | send | Node1 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:08,963 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node2 2017-08-28 15:18:08,976 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node2 2017-08-28 15:18:08,976 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node3 2017-08-28 15:18:08,977 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node3 2017-08-28 15:18:08,983 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node4 2017-08-28 15:18:08,984 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node2","viewNo":1}' to Node4 2017-08-28 15:18:08,984 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:08,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:08,994 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:08,995 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:18:09,011 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:09,019 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,022 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'viewNo': 1}, 'Node4') 2017-08-28 15:18:09,022 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,023 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,027 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'viewNo': 1}, 'Node3') 2017-08-28 15:18:09,035 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,036 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,037 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node2', 'viewNo': 1}, 'Node2') 2017-08-28 15:18:09,047 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,048 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node4') 2017-08-28 15:18:09,049 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node3') 2017-08-28 15:18:09,054 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node2') 2017-08-28 15:18:09,068 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,069 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 1 ViewChangeDone messages 2017-08-28 15:18:09,069 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:09,080 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,080 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (3 >= 3) 2017-08-28 15:18:09,081 | DEBUG | primary_selector.py ( 208) | has_view_change_from_primary | Node1 has not received ViewChangeDone from the next primary Node2 2017-08-28 15:18:09,093 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:18:09,093 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-08-28 15:18:09,094 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (4 >= 3) 2017-08-28 15:18:09,098 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node2 2017-08-28 15:18:09,098 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node2 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:18:09,099 | DEBUG | primary_selector.py ( 280) | _startSelection | Node1 starting selection 2017-08-28 15:18:09,109 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node2:0 for instance 0 (view 1) 2017-08-28 15:18:09,109 | INFO | node.py ( 503) | start_participating | Node1 started participating 2017-08-28 15:18:09,110 | DEBUG | replica.py ( 406) | primaryName | Node1:0 setting primaryName for view no 1 to: Node2:0 2017-08-28 15:18:09,110 | DEBUG | replica.py (1678) | _gc | Node1:0 cleaning up till (0, 6) 2017-08-28 15:18:09,113 | DEBUG | replica.py (1693) | _gc | Node1:0 found 6 3-phase keys to clean 2017-08-28 15:18:09,123 | DEBUG | replica.py (1695) | _gc | Node1:0 found 6 request keys to clean 2017-08-28 15:18:09,124 | DEBUG | replica.py ( 311) | h | Node1:0 set watermarks as 0 300 2017-08-28 15:18:09,125 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,126 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:1 selected primary Node3:1 for instance 1 (view 1) 2017-08-28 15:18:09,131 | DEBUG | replica.py ( 406) | primaryName | Node1:1 setting primaryName for view no 1 to: Node3:1 2017-08-28 15:18:09,132 | DEBUG | replica.py (1678) | _gc | Node1:1 cleaning up till (0, 6) 2017-08-28 15:18:09,132 | DEBUG | replica.py (1693) | _gc | Node1:1 found 6 3-phase keys to clean 2017-08-28 15:18:09,142 | DEBUG | replica.py (1695) | _gc | Node1:1 found 6 request keys to clean 2017-08-28 15:18:09,143 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933459424488) from previous checkpoints 2017-08-28 15:18:09,143 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933471684615) from previous checkpoints 2017-08-28 15:18:09,143 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933484172953) from previous checkpoints 2017-08-28 15:18:09,149 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933200605368) from previous checkpoints 2017-08-28 15:18:09,149 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933478716086) from previous checkpoints 2017-08-28 15:18:09,150 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933466149129) from previous checkpoints 2017-08-28 15:18:09,150 | DEBUG | replica.py ( 311) | h | Node1:1 set watermarks as 0 300 2017-08-28 15:18:09,163 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node1:1 Setting last ordered for non-master as (0, 6) 2017-08-28 15:18:09,164 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (1, 0) 2017-08-28 15:18:09,164 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:18:09,197 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 2.038 seconds 2017-08-28 15:18:12,752 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:12,753 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,761 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:12,763 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:12,771 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:14,016 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 46 2017-08-28 15:18:14,016 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:14,016 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:14,017 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:14,017 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:14,017 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:18:14,018 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:14,018 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-08-28 15:18:15,642 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:18:15,669 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933495609306 2017-08-28 15:18:15,669 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,670 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,671 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,681 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,690 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933495609306 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:15,690 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:15,695 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since not finalised 2017-08-28 15:18:15,705 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:15,714 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node2 2017-08-28 15:18:15,726 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node2 2017-08-28 15:18:15,727 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node3 2017-08-28 15:18:15,733 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node3 2017-08-28 15:18:15,742 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node4 2017-08-28 15:18:15,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}}' to Node4 2017-08-28 15:18:15,750 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:15,773 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,782 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,786 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}}, 'Node4') 2017-08-28 15:18:15,786 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,794 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,795 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,811 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}}, 'Node2') 2017-08-28 15:18:15,814 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,821 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,822 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,832 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) to 0 replicas 2017-08-28 15:18:15,832 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}} 2017-08-28 15:18:15,849 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,849 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since already forwarded 2017-08-28 15:18:15,876 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:15,877 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933495609306 2017-08-28 15:18:15,893 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,894 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}}, 'Node3') 2017-08-28 15:18:15,900 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,906 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933495609306}} 2017-08-28 15:18:15,912 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} 2017-08-28 15:18:15,912 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306} to its replicas since already forwarded 2017-08-28 15:18:15,952 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.330 seconds 2017-08-28 15:18:15,998 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:16,011 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:18:16,012 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPREPARE', 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'ppTime': 1503933495, 'discarded': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,012 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:18:16,025 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(1, 1) from Node3:1 at 249792.17395849 2017-08-28 15:18:16,040 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(1, 1) at 249792.18890585 2017-08-28 15:18:16,040 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:18:16,040 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,040 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,052 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933495, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} with state root None 2017-08-28 15:18:16,052 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(1, 1) 2017-08-28 15:18:16,080 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:16,081 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:16,082 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:18:16,082 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'ppTime': 1503933495, 'op': 'PREPREPARE', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'discarded': 0, 'viewNo': 1, 'ledgerId': 0}, 'Node2') 2017-08-28 15:18:16,082 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933495609306]], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:18:16,111 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node2 2017-08-28 15:18:16,112 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node2 2017-08-28 15:18:16,119 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node3 2017-08-28 15:18:16,123 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node3 2017-08-28 15:18:16,127 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node4 2017-08-28 15:18:16,128 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":null,"ppTime":1503933495,"ppSeqNo":1,"instId":1,"txnRootHash":null,"viewNo":1}' to Node4 2017-08-28 15:18:16,129 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 1) from Node2:0 at 249792.278259472 2017-08-28 15:18:16,145 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:18:16,148 | WARNING | replica.py ( 655) | processReqDuringBatch | Node1:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'signature': 'NVcKE1L2pvGYKpxuTQFjJqexBezqAowzsDWZpRQWyXUn7iMhpidKKAN97bkmG5TFLMph8xZgq3Wa3wgeVcSqq4e', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933495609306}, will reject 2017-08-28 15:18:16,164 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 1) at 249792.313237682 2017-08-28 15:18:16,164 | DEBUG | replica.py (2133) | send | Node1:0 sending Prepare 2017-08-28 15:18:16,167 | TRACE | replica.py (2134) | send | Node1:0 sending PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,168 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,177 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933495, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:16,180 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 1) 2017-08-28 15:18:16,182 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:16,199 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,200 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933495, 'op': 'PREPARE', 'ppSeqNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1}, 'Node2') 2017-08-28 15:18:16,200 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,233 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)","op":"REJECT","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933495609306}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:16,233 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:16,234 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 1) from Node2:1 2017-08-28 15:18:16,234 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(1, 1) at 249792.38316865 2017-08-28 15:18:16,234 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:18:16,250 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,250 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:18:16,251 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,268 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:16,269 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,279 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933495, 'op': 'PREPARE', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,283 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,297 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node2 2017-08-28 15:18:16,298 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node2 2017-08-28 15:18:16,302 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node3 2017-08-28 15:18:16,303 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node3 2017-08-28 15:18:16,311 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node4 2017-08-28 15:18:16,312 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933495,"ppSeqNo":1,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node4 2017-08-28 15:18:16,326 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:16,326 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 1) from Node4:1 2017-08-28 15:18:16,327 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,335 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,337 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:16,345 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,346 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:18:16,346 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,356 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,356 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933495, 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,357 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,362 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,362 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1, 'instId': 1}, 'Node2') 2017-08-28 15:18:16,363 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,382 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node2 2017-08-28 15:18:16,391 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node2 2017-08-28 15:18:16,392 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,392 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,404 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,404 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,405 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 1) from Node3:0 2017-08-28 15:18:16,415 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 1) at 249792.564426737 2017-08-28 15:18:16,416 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:18:16,416 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,416 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:18:16,424 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,425 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 1) from Node4:0 2017-08-28 15:18:16,428 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '2763ecc2638129b576451908993119a00faa541740461e4adba7d3a17b793761', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933495, 'ppSeqNo': 1} 2017-08-28 15:18:16,428 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 1) 2017-08-28 15:18:16,428 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 1) from Node2:1 2017-08-28 15:18:16,430 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:18:16,441 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,443 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:16,453 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,459 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 1, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,459 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,460 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,460 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node3') 2017-08-28 15:18:16,465 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:16,483 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:16,492 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 1) from Node4:1 2017-08-28 15:18:16,492 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:18:16,492 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (1, 1) 2017-08-28 15:18:16,499 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (1, 1) 2017-08-28 15:18:16,499 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:18:16,500 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 1, 'ppTime': 1503933495, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:18:16,510 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (1, 1) 2017-08-28 15:18:16,510 | DEBUG | replica.py (1540) | _newCheckpointState | Node1:1 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,510 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,511 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:18:16,520 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:16,527 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,528 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node3') 2017-08-28 15:18:16,528 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,530 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,537 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1, 'instId': 0}, 'Node2') 2017-08-28 15:18:16,538 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,555 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node2 2017-08-28 15:18:16,562 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node2 2017-08-28 15:18:16,562 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,562 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node3 2017-08-28 15:18:16,576 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,577 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":1}' to Node4 2017-08-28 15:18:16,584 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:18:16,584 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 1) from Node3:0 2017-08-28 15:18:16,594 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:18:16,595 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,597 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 1) from Node2:0 2017-08-28 15:18:16,597 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:18:16,598 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (1, 1) 2017-08-28 15:18:16,598 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (1, 1) 2017-08-28 15:18:16,598 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:18:16,613 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 1, 'ppTime': 1503933495, 'instId': 0, 'reqIdr': [], 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:18:16,613 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (1, 1) 2017-08-28 15:18:16,613 | DEBUG | replica.py (1540) | _newCheckpointState | Node1:0 adding new checkpoint state for (1, 100) 2017-08-28 15:18:16,614 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 1) 2017-08-28 15:18:16,626 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:16,634 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,635 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 0, 'ppSeqNo': 1}, 'Node4') 2017-08-28 15:18:16,635 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:16,661 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 1 1 of 0 requests 2017-08-28 15:18:16,686 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:16,695 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:16,695 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 1, 'viewNo': 1, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:18:16,747 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.758 seconds 2017-08-28 15:18:23,201 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:18:23,233 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1503933503177919 2017-08-28 15:18:23,233 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,237 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,249 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,250 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,267 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating V4SGRU86Z58d6TV7PBUe6f request 1503933503177919 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:18:23,268 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:23,268 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since not finalised 2017-08-28 15:18:23,276 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,288 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node2 2017-08-28 15:18:23,292 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node2 2017-08-28 15:18:23,300 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node3 2017-08-28 15:18:23,313 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node3 2017-08-28 15:18:23,314 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node4 2017-08-28 15:18:23,324 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}}' to Node4 2017-08-28 15:18:23,326 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,349 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,350 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,358 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}}, 'Node4') 2017-08-28 15:18:23,359 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,370 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,371 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,383 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) to 0 replicas 2017-08-28 15:18:23,410 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:23,416 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,425 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,426 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}}, 'Node2') 2017-08-28 15:18:23,433 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,434 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1503933503177919 2017-08-28 15:18:23,446 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,452 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}}, 'Node3') 2017-08-28 15:18:23,454 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,458 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}} 2017-08-28 15:18:23,477 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,477 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since already forwarded 2017-08-28 15:18:23,487 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933503177919}} 2017-08-28 15:18:23,489 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} 2017-08-28 15:18:23,489 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919} to its replicas since already forwarded 2017-08-28 15:18:23,533 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.334 seconds 2017-08-28 15:18:23,569 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,576 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 0} 2017-08-28 15:18:23,576 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPREPARE', 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'ppTime': 1503933503, 'discarded': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:23,588 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 0} 2017-08-28 15:18:23,597 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(1, 2) from Node3:1 at 249799.74592322 2017-08-28 15:18:23,608 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(1, 2) at 249799.756908999 2017-08-28 15:18:23,618 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:18:23,618 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,618 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,619 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933503, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 2, 'ledgerId': 0} with state root None 2017-08-28 15:18:23,627 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(1, 2) 2017-08-28 15:18:23,628 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,629 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 0} 2017-08-28 15:18:23,641 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'ppTime': 1503933503, 'op': 'PREPREPARE', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'discarded': 0, 'viewNo': 1, 'ledgerId': 0}, 'Node2') 2017-08-28 15:18:23,641 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1503933503177919]], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 0} 2017-08-28 15:18:23,663 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:23,667 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 2) from Node2:0 at 249799.816338645 2017-08-28 15:18:23,668 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 0} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:18:23,679 | WARNING | replica.py ( 655) | processReqDuringBatch | Node1:0 encountered exception UnauthorizedClientRequest('V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node',) while processing Request: {'signature': '5dpELXGTPaWRvm1tdzq9yRHfTHjW6CVVfgStKsxKyjpLm7VYS38WKpyFpTgfY82JRrqAKHEzV3WMAtiFQDXE4YYa', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1503933503177919}, will reject 2017-08-28 15:18:23,680 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 2) at 249799.829612723 2017-08-28 15:18:23,690 | DEBUG | replica.py (2133) | send | Node1:0 sending Prepare 2017-08-28 15:18:23,690 | TRACE | replica.py (2134) | send | Node1:0 sending PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,691 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,699 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933503, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 2, 'ledgerId': 0} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:18:23,702 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 2) 2017-08-28 15:18:23,712 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,713 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,721 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933503, 'op': 'PREPARE', 'ppSeqNo': 2, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1}, 'Node2') 2017-08-28 15:18:23,721 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,740 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node2 2017-08-28 15:18:23,741 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node2 2017-08-28 15:18:23,747 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node3 2017-08-28 15:18:23,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node3 2017-08-28 15:18:23,748 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node4 2017-08-28 15:18:23,748 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":null,"ppTime":1503933503,"ppSeqNo":2,"instId":1,"txnRootHash":null,"viewNo":1}' to Node4 2017-08-28 15:18:23,766 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"reason":"client request invalid: UnauthorizedClientRequest(\'V4SGRU86Z58d6TV7PBUe6f is not a steward so cannot add a new node\',)","op":"REJECT","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1503933503177919}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:18:23,773 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:23,774 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 2) from Node2:1 2017-08-28 15:18:23,774 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(1, 2) at 249799.923181757 2017-08-28 15:18:23,775 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:18:23,792 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:23,792 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:18:23,792 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,793 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,803 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,811 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933503, 'op': 'PREPARE', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:23,813 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,841 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node2 2017-08-28 15:18:23,846 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node2 2017-08-28 15:18:23,846 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node3 2017-08-28 15:18:23,868 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node3 2017-08-28 15:18:23,871 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node4 2017-08-28 15:18:23,881 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"digest":"db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933503,"ppSeqNo":2,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}' to Node4 2017-08-28 15:18:23,883 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:23,892 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 2) from Node4:1 2017-08-28 15:18:23,892 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,893 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,894 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:23,908 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,912 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2}, 'Node3') 2017-08-28 15:18:23,912 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,929 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node2 2017-08-28 15:18:23,930 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node2 2017-08-28 15:18:23,936 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:23,936 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:23,937 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:23,948 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:23,949 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 2) from Node3:0 2017-08-28 15:18:23,956 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 2) at 249800.104907426 2017-08-28 15:18:23,956 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:18:23,956 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:23,957 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:18:23,968 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:23,973 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:18:23,974 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,987 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933503, 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:23,988 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:23,988 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:23,999 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1, 'instId': 1}, 'Node2') 2017-08-28 15:18:24,000 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:24,009 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:24,017 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:24,018 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:24,028 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:18:24,029 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:18:24,031 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 2) from Node4:0 2017-08-28 15:18:24,038 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'db7ef9a4953ffebad24e1d3f85290d1df59ed3cc46b20462f1ecd5582147e2d5', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933503, 'ppSeqNo': 2} 2017-08-28 15:18:24,038 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 2) 2017-08-28 15:18:24,050 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 2) from Node2:1 2017-08-28 15:18:24,051 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:18:24,057 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,058 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:24,066 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:24,066 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node3') 2017-08-28 15:18:24,075 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:24,077 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 48 2017-08-28 15:18:24,087 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:24,088 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:24,091 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:24,091 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:24,092 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:24,099 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:24,100 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-08-28 15:18:24,109 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node2 2017-08-28 15:18:24,109 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node2 2017-08-28 15:18:24,110 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:24,111 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node3 2017-08-28 15:18:24,121 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:24,121 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":2}' to Node4 2017-08-28 15:18:24,129 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 2) from Node3:1 2017-08-28 15:18:24,139 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:18:24,140 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (1, 2) 2017-08-28 15:18:24,140 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (1, 2) 2017-08-28 15:18:24,140 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:18:24,141 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 2, 'ppTime': 1503933503, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:18:24,146 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (1, 2) 2017-08-28 15:18:24,147 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,156 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:18:24,167 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:24,167 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 1, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,168 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} 2017-08-28 15:18:24,168 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,179 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node3') 2017-08-28 15:18:24,180 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,180 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,180 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1, 'instId': 0}, 'Node2') 2017-08-28 15:18:24,185 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,202 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:18:24,214 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 2) from Node3:0 2017-08-28 15:18:24,215 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:18:24,215 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,221 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 2) from Node2:0 2017-08-28 15:18:24,224 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:18:24,228 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (1, 2) 2017-08-28 15:18:24,228 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (1, 2) 2017-08-28 15:18:24,229 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:18:24,234 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 2, 'ppTime': 1503933503, 'instId': 0, 'reqIdr': [], 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:18:24,237 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (1, 2) 2017-08-28 15:18:24,238 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 2) 2017-08-28 15:18:24,247 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:18:24,248 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:24,252 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,252 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 0, 'ppSeqNo': 2}, 'Node4') 2017-08-28 15:18:24,262 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} 2017-08-28 15:18:24,287 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 1 2 of 0 requests 2017-08-28 15:18:24,288 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:18:24,288 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:24,290 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 2, 'viewNo': 1, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:18:24,322 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.755 seconds 2017-08-28 15:18:28,027 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:28,028 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,031 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:28,037 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:28,049 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:34,129 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 49 2017-08-28 15:18:34,138 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:34,141 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:34,147 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:34,155 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:34,156 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:34,156 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:34,164 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-08-28 15:18:37,671 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:37,672 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,673 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:37,685 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:37,697 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:39,026 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:39,037 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:39,038 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:39,052 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:18:42,777 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:42,778 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,786 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:42,789 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:42,802 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:18:44,190 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 50 2017-08-28 15:18:44,193 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:44,196 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:44,197 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:44,214 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:44,215 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:44,215 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:44,215 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 51 to run in 10 seconds 2017-08-28 15:18:52,336 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:52,338 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,339 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:18:52,353 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:18:52,362 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:18:53,275 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 44 2017-08-28 15:18:53,279 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:18:53,279 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:53,287 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 52 to run in 60 seconds 2017-08-28 15:18:53,288 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-08-28 15:18:53,288 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:18:53,288 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-08-28 15:18:54,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:18:54,043 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:18:54,048 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:18:54,048 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:18:54,220 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 51 2017-08-28 15:18:54,221 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:18:54,221 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:18:54,223 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:18:54,228 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:18:54,228 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:18:54,232 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:18:54,248 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-08-28 15:18:58,025 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:18:58,030 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,031 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:18:58,048 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:18:58,051 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:04,248 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 53 2017-08-28 15:19:04,248 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:04,269 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:04,269 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:04,269 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:04,270 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:04,270 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:04,271 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-08-28 15:19:07,231 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action _check_view_change_completed with id 47 2017-08-28 15:19:07,243 | DEBUG | node.py ( 993) | _check_view_change_completed | Node1 running the scheduled check for view change completion 2017-08-28 15:19:07,244 | DEBUG | node.py ( 995) | _check_view_change_completed | Node1 already completion view change 2017-08-28 15:19:07,540 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:07,548 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,552 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:07,561 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:07,576 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:09,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:09,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:09,062 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:09,062 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:19:12,785 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:12,787 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,790 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:12,802 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:12,811 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:14,276 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 54 2017-08-28 15:19:14,303 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:14,303 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:14,303 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:14,304 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:14,304 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:14,308 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:14,308 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-08-28 15:19:22,353 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:22,353 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,364 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:22,366 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:22,377 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:24,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:24,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:24,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:24,061 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:19:24,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 55 2017-08-28 15:19:24,352 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:24,352 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:24,362 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:24,363 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:24,363 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:24,367 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:24,368 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-08-28 15:19:28,029 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:28,037 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,038 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:28,048 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:28,064 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:34,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 56 2017-08-28 15:19:34,406 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:34,406 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:34,407 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:34,407 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:34,408 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:34,409 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:34,409 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-08-28 15:19:37,538 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:37,552 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,552 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:37,563 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:37,570 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:39,062 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:39,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:39,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:39,065 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:19:42,812 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:42,820 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,820 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:42,821 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:42,822 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:19:44,423 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 57 2017-08-28 15:19:44,428 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:44,435 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:44,436 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:44,436 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:44,436 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:44,449 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:44,449 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-08-28 15:19:52,365 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:52,370 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,371 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:19:52,390 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:19:52,390 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:19:53,308 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 52 2017-08-28 15:19:53,309 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:19:53,309 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:53,309 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds 2017-08-28 15:19:53,330 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 8 2017-08-28 15:19:53,331 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:19:53,335 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-08-28 15:19:54,076 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:19:54,076 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:19:54,078 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:19:54,092 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:19:54,470 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 58 2017-08-28 15:19:54,473 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:19:54,474 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:19:54,474 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:19:54,474 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:19:54,474 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:19:54,475 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:19:54,475 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-08-28 15:19:58,117 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:19:58,117 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,118 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:19:58,120 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:19:58,136 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:04,486 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 60 2017-08-28 15:20:04,489 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:04,489 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:04,489 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:04,489 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:04,490 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:04,490 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:04,490 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 61 to run in 10 seconds 2017-08-28 15:20:07,678 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:07,678 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,679 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:07,690 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:07,694 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:09,096 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:09,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:09,104 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:09,105 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:20:12,813 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:12,813 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,820 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:12,822 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:12,841 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:14,501 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 61 2017-08-28 15:20:14,502 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:14,502 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:14,502 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:14,502 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:14,502 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:14,503 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:14,503 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-08-28 15:20:22,401 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:22,402 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,402 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:22,419 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:22,423 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:24,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:24,119 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:24,120 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:24,126 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:20:24,521 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 62 2017-08-28 15:20:24,531 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:24,535 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:24,539 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:24,539 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:24,543 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:24,547 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:24,548 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-08-28 15:20:28,089 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:28,090 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,103 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:28,116 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:28,116 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:34,566 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 63 2017-08-28 15:20:34,582 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:34,583 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:34,588 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:34,588 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:34,588 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:34,589 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:34,592 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-08-28 15:20:37,622 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:37,628 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,636 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:37,655 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:37,655 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:39,127 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:39,130 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:39,130 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:39,131 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:20:42,848 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:42,865 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,869 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:42,870 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:42,879 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:20:44,606 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 64 2017-08-28 15:20:44,608 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:44,608 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:44,609 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:44,609 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:44,610 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:44,610 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:44,610 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-08-28 15:20:52,425 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:52,426 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,436 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:20:52,437 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:20:52,438 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:20:53,323 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 59 2017-08-28 15:20:53,323 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:20:53,324 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:53,324 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 66 to run in 60 seconds 2017-08-28 15:20:53,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 9 2017-08-28 15:20:53,364 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:20:53,364 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-08-28 15:20:54,161 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:20:54,168 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:20:54,168 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:20:54,169 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:20:54,626 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 65 2017-08-28 15:20:54,637 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:20:54,637 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:20:54,637 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:20:54,639 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:20:54,640 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:20:54,640 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:20:54,640 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-08-28 15:20:58,000 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:20:58,007 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,008 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:20:58,028 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:20:58,028 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:04,665 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 67 2017-08-28 15:21:04,666 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:04,666 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:04,667 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:04,673 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:04,673 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:04,675 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:04,675 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-08-28 15:21:07,547 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:07,551 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,555 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:07,558 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:07,568 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:09,185 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:09,186 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:09,186 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:09,187 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:21:12,893 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:12,906 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,907 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:12,909 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:12,924 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:14,695 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 68 2017-08-28 15:21:14,699 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:14,700 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:14,700 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:14,700 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:14,700 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:14,700 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:14,700 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-08-28 15:21:22,462 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:22,463 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,473 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:22,484 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:22,484 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:24,189 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:24,199 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:24,203 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:24,217 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:21:24,704 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 69 2017-08-28 15:21:24,705 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:24,705 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:24,705 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:24,712 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:24,712 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:24,713 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:24,713 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-08-28 15:21:28,131 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:28,137 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,137 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:28,145 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:28,154 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:34,721 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 70 2017-08-28 15:21:34,722 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:34,735 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:34,738 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:34,738 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:34,739 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:34,739 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:34,744 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-08-28 15:21:37,645 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:37,646 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,646 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:37,660 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:37,672 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:39,220 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:39,220 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:39,220 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:39,224 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:21:42,911 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:42,918 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,918 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:42,929 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:42,944 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:21:44,759 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 71 2017-08-28 15:21:44,770 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:44,772 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:44,773 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:44,773 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:44,774 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:44,775 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:44,775 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-08-28 15:21:52,491 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:52,498 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,498 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:21:52,512 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:21:52,512 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:21:53,334 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 66 2017-08-28 15:21:53,335 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:21:53,335 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:53,338 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 73 to run in 60 seconds 2017-08-28 15:21:53,393 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 10 2017-08-28 15:21:53,401 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:21:53,402 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-08-28 15:21:54,248 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:21:54,248 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:21:54,248 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:21:54,267 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:21:54,797 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 72 2017-08-28 15:21:54,812 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:21:54,812 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:21:54,812 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:21:54,813 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:21:54,815 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:21:54,816 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:21:54,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-08-28 15:21:58,133 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:21:58,143 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,144 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:21:58,158 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:21:58,176 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:04,837 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 74 2017-08-28 15:22:04,841 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:04,848 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:04,855 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:04,855 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:04,862 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:04,865 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:04,868 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-08-28 15:22:07,917 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:07,918 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,919 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:07,925 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:07,951 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:09,273 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:09,277 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:09,279 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:09,295 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:22:12,905 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:12,905 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,907 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:12,909 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:12,927 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:14,872 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 75 2017-08-28 15:22:14,888 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:14,888 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:14,888 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:14,889 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:14,889 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:14,899 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:14,899 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-08-28 15:22:22,564 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:22,565 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,565 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:22,589 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:22,591 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:24,275 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:24,276 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:24,276 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:24,284 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:22:24,899 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 76 2017-08-28 15:22:24,900 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:24,900 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:24,910 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:24,910 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:24,911 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:24,914 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:24,914 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 77 to run in 10 seconds 2017-08-28 15:22:28,055 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:28,062 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,062 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:28,064 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:28,083 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:34,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 77 2017-08-28 15:22:34,920 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:34,921 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:34,943 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:34,948 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:34,948 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:34,948 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:34,949 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-08-28 15:22:37,771 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:37,774 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,774 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:37,792 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:37,792 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:39,292 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:39,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:39,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:39,307 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:22:42,953 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:42,956 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,956 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:42,983 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:42,988 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:22:44,953 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 78 2017-08-28 15:22:44,955 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:44,959 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:44,963 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:44,966 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:44,967 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:44,975 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:44,976 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-08-28 15:22:52,560 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:52,562 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,574 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:22:52,583 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:22:52,596 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:22:53,348 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 73 2017-08-28 15:22:53,348 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:22:53,349 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:53,349 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 80 to run in 60 seconds 2017-08-28 15:22:53,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 11 2017-08-28 15:22:53,415 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:22:53,416 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-08-28 15:22:54,320 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:22:54,321 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:22:54,323 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:22:54,323 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:22:55,005 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 79 2017-08-28 15:22:55,024 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:22:55,025 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:22:55,025 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:22:55,027 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:22:55,030 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:22:55,030 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:22:55,031 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-08-28 15:22:58,197 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:22:58,201 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,201 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:22:58,202 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:22:58,202 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:05,060 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 81 2017-08-28 15:23:05,060 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:05,074 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:05,074 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:05,075 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:05,075 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:05,080 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:05,080 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-08-28 15:23:07,725 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:07,726 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,743 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:07,745 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:07,745 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:09,332 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:09,341 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:09,344 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:09,345 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:23:12,979 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:12,980 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:12,995 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:13,006 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:13,021 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:15,108 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 82 2017-08-28 15:23:15,128 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:15,129 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:15,129 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:15,131 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:15,132 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:15,137 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:15,137 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-08-28 15:23:22,597 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:22,598 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,598 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:22,611 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:22,624 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:24,336 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:24,352 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:24,352 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:24,353 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:23:25,158 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 83 2017-08-28 15:23:25,168 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:25,171 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:25,172 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:25,179 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:25,180 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:25,180 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:25,180 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-08-28 15:23:28,091 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:28,092 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,092 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:28,094 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:28,114 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:35,220 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 84 2017-08-28 15:23:35,226 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:35,229 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:35,229 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:35,229 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:35,230 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:35,230 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:35,231 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 85 to run in 10 seconds 2017-08-28 15:23:37,825 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:37,836 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,836 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:37,838 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:37,855 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:39,355 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:39,360 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:39,360 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:39,362 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:23:42,973 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:42,975 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:42,979 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:42,981 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:42,998 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:23:45,251 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 85 2017-08-28 15:23:45,262 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:45,263 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:45,263 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:45,267 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:45,268 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:45,268 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:45,268 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-08-28 15:23:52,566 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:52,566 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,567 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:23:52,573 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:23:52,578 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:23:53,353 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 80 2017-08-28 15:23:53,354 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:23:53,354 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:53,354 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 87 to run in 60 seconds 2017-08-28 15:23:53,418 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 12 2017-08-28 15:23:53,436 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:23:53,437 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-08-28 15:23:54,376 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:23:54,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:23:54,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:23:54,388 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:23:55,295 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 86 2017-08-28 15:23:55,295 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:23:55,296 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:23:55,298 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:23:55,298 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:23:55,299 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:23:55,299 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:23:55,299 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-08-28 15:23:58,158 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:23:58,169 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,169 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:23:58,172 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:23:58,180 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:05,311 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 88 2017-08-28 15:24:05,326 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:05,327 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:05,327 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:05,331 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:05,342 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:05,343 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:05,343 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-08-28 15:24:07,725 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:07,725 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,726 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:07,727 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:07,742 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:09,379 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:09,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:09,384 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:09,385 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:24:12,987 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:12,988 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:12,988 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:12,998 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:12,999 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:15,359 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 89 2017-08-28 15:24:15,363 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:15,363 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:15,364 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:15,364 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:15,364 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:15,367 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:15,367 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-08-28 15:24:22,616 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:22,624 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,624 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:22,626 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:22,642 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:24,379 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:24,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:24,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:24,384 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:24:25,373 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 90 2017-08-28 15:24:25,383 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:25,384 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:25,384 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:25,386 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:25,386 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:25,386 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:25,386 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-08-28 15:24:28,183 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:28,184 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,195 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:28,206 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:28,206 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:35,409 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 91 2017-08-28 15:24:35,423 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:35,424 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:35,424 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:35,424 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:35,426 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:35,427 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:35,427 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-08-28 15:24:37,793 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:37,804 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,804 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:37,806 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:37,806 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:39,396 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:39,397 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:39,398 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:39,400 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:24:43,015 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:43,016 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,023 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:43,025 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:43,025 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:24:45,448 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 92 2017-08-28 15:24:45,451 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:45,457 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:45,457 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:45,458 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:45,458 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:45,460 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:45,460 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 93 to run in 10 seconds 2017-08-28 15:24:52,610 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:52,612 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,625 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:24:52,639 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:24:52,647 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:24:53,358 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 87 2017-08-28 15:24:53,359 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:24:53,363 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:53,367 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 94 to run in 60 seconds 2017-08-28 15:24:53,457 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 13 2017-08-28 15:24:53,462 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:24:53,463 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-08-28 15:24:54,421 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:24:54,433 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:24:54,433 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:24:54,434 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:24:55,474 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 93 2017-08-28 15:24:55,479 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:24:55,490 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:24:55,490 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:24:55,491 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:24:55,491 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:24:55,495 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:24:55,499 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-08-28 15:24:58,230 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:24:58,231 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,241 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:24:58,253 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:24:58,261 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:05,501 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 95 2017-08-28 15:25:05,502 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:05,502 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:05,503 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:05,503 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:05,504 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:05,508 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:25:05,520 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-08-28 15:25:07,888 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:07,892 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:07,892 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:07,922 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:07,923 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:09,446 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:09,454 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:09,464 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:09,472 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:25:13,026 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:13,028 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,043 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:13,047 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:13,049 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:15,529 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 96 2017-08-28 15:25:15,530 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:15,530 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:15,538 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:15,541 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:15,541 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:15,542 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:25:15,557 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-08-28 15:25:22,609 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:22,616 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,616 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:22,621 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:22,637 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:24,454 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:24,460 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:24,460 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:24,461 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:25:25,576 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 97 2017-08-28 15:25:25,576 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:25,583 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:25,586 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:25,586 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:25,587 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:25,587 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:25:25,589 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-08-28 15:25:28,347 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:28,356 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,356 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:28,361 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:28,376 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:35,620 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 98 2017-08-28 15:25:35,623 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:35,624 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:35,624 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:35,624 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:35,632 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:35,641 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:25:35,641 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 99 to run in 10 seconds 2017-08-28 15:25:37,912 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:37,916 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,916 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:37,939 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:37,940 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:39,486 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:39,495 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:39,498 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:39,498 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:25:43,087 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:43,088 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,095 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' 2017-08-28 15:25:43,098 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' has been removed 2017-08-28 15:25:43,099 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'BEhc!q7/{^9)vgy&p$f!kx.sTfa9gxeT9(BqKVil' no longer available 2017-08-28 15:25:45,664 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 99 2017-08-28 15:25:45,665 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:45,665 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:45,665 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:45,666 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:45,667 | TRACE | monitor.py ( 378) | isMasterAvgReqLatencyTooHigh | Node1 found master had no record yet for V4SGRU86Z58d6TV7PBUe6f 2017-08-28 15:25:45,667 | DEBUG | node.py (1976) | checkPerformance | Node1's master has higher performance than backups 2017-08-28 15:25:45,684 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 100 to run in 10 seconds 2017-08-28 15:25:47,497 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:25:47,498 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on request 1503933947440487 2017-08-28 15:25:47,512 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,512 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,520 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,531 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,536 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating 6XiqM9DwYDgFnmyQQBFr72 request 1503933947440487 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:47,536 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:47,545 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since not finalised 2017-08-28 15:25:47,554 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,560 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node2 2017-08-28 15:25:47,560 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node2 2017-08-28 15:25:47,567 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node3 2017-08-28 15:25:47,568 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node3 2017-08-28 15:25:47,568 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node4 2017-08-28 15:25:47,568 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}}' to Node4 2017-08-28 15:25:47,570 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:47,612 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,612 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,613 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}}, 'Node2') 2017-08-28 15:25:47,622 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,631 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,636 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,636 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}}, 'Node4') 2017-08-28 15:25:47,637 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,648 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}} 2017-08-28 15:25:47,652 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,658 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) to 0 replicas 2017-08-28 15:25:47,659 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,670 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,677 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since already forwarded 2017-08-28 15:25:47,699 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:47,719 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated 6XiqM9DwYDgFnmyQQBFr72 signature on propagate request 1503933947440487 2017-08-28 15:25:47,719 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,728 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}}, 'Node3') 2017-08-28 15:25:47,728 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,735 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933947440487}} 2017-08-28 15:25:47,740 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} 2017-08-28 15:25:47,752 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487} to its replicas since already forwarded 2017-08-28 15:25:47,778 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.298 seconds 2017-08-28 15:25:47,831 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:47,832 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 0} 2017-08-28 15:25:47,835 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'ppTime': 1503933947, 'op': 'PREPREPARE', 'instId': 0, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'discarded': 0, 'viewNo': 1, 'ledgerId': 0}, 'Node2') 2017-08-28 15:25:47,843 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 0} 2017-08-28 15:25:47,844 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 0} 2017-08-28 15:25:47,844 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPREPARE', 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'ppTime': 1503933947, 'discarded': 1, 'txnRootHash': None, 'ledgerId': 0, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:47,848 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [['6XiqM9DwYDgFnmyQQBFr72', 1503933947440487]], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 0} 2017-08-28 15:25:47,869 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 3) from Node2:0 at 250244.018526707 2017-08-28 15:25:47,874 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 0} is b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4', 5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3 2017-08-28 15:25:47,885 | WARNING | replica.py ( 655) | processReqDuringBatch | Node1:0 encountered exception UnauthorizedClientRequest('6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node',) while processing Request: {'signature': '2cxVqZd6QCJNybeMAAUrNEwS5EdtE6U3twEaVCsF3hU8fpFH2KWXuMbsbS4yWeUFD2hpsecnbochJ5xFu39c816V', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': '6XiqM9DwYDgFnmyQQBFr72', 'reqId': 1503933947440487}, will reject 2017-08-28 15:25:47,892 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 3) at 250244.041144707 2017-08-28 15:25:47,892 | DEBUG | replica.py (2133) | send | Node1:0 sending Prepare 2017-08-28 15:25:47,903 | TRACE | replica.py (2134) | send | Node1:0 sending PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,904 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,904 | DEBUG | replica.py ( 612) | trackBatches | Node1:0 tracking batch for PREPREPARE{'ppTime': 1503933947, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 1, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 3, 'ledgerId': 0} with state root b'\xf483\x87\xf9\x9cf\x8e7\x8a\xc1\xf2Ei\x1a\x83\x93\x8a\xcb\x88\xf9i\x1f\xd5\xcb\x0f\x9a\xc0\xf2T\x14\xb4' 2017-08-28 15:25:47,912 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 3) 2017-08-28 15:25:47,912 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(1, 3) from Node3:1 at 250244.061312134 2017-08-28 15:25:47,913 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(1, 3) at 250244.061830519 2017-08-28 15:25:47,918 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:25:47,919 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,925 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,925 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933947, 'stateRootHash': None, 'viewNo': 1, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 3, 'ledgerId': 0} with state root None 2017-08-28 15:25:47,926 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(1, 3) 2017-08-28 15:25:47,934 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"reason":"client request invalid: UnauthorizedClientRequest(\'6XiqM9DwYDgFnmyQQBFr72 is not a steward so cannot add a new node\',)","op":"REJECT","identifier":"6XiqM9DwYDgFnmyQQBFr72","reqId":1503933947440487}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:47,961 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:47,961 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:47,962 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:47,962 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,975 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933947, 'op': 'PREPARE', 'ppSeqNo': 3, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1}, 'Node2') 2017-08-28 15:25:47,977 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:47,996 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:47,997 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}']) 2017-08-28 15:25:47,997 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' 2017-08-28 15:25:48,010 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:48,011 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:48,019 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}']) 2017-08-28 15:25:48,020 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' 2017-08-28 15:25:48,025 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:48,025 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:48,025 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppTime":1503933947,"ppSeqNo":3,"instId":0,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":1}', b'{"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6","op":"PREPARE","stateRootHash":null,"ppTime":1503933947,"ppSeqNo":3,"instId":1,"txnRootHash":null,"viewNo":1}']) 2017-08-28 15:25:48,025 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' 2017-08-28 15:25:48,025 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":\\"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m\\",\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":0,\\"txnRootHash\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"viewNo\\":1}","{\\"digest\\":\\"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6\\",\\"op\\":\\"PREPARE\\",\\"stateRootHash\\":null,\\"ppTime\\":1503933947,\\"ppSeqNo\\":3,\\"instId\\":1,\\"txnRootHash\\":null,\\"viewNo\\":1}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:48,027 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 3) from Node2:1 2017-08-28 15:25:48,027 | DEBUG | replica.py ( 953) | doCommit | Node1:1 Sending COMMIT(1, 3) at 250244.176049296 2017-08-28 15:25:48,027 | DEBUG | replica.py (2133) | send | Node1:1 sending Commit 2017-08-28 15:25:48,058 | TRACE | replica.py (2134) | send | Node1:1 sending COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,058 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:25:48,058 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,069 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:48,070 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,095 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3}, 'Node3') 2017-08-28 15:25:48,096 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,096 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}']} 2017-08-28 15:25:48,106 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:48,107 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"PREPARE","ppTime":1503933947,"stateRootHash":"HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m","ppSeqNo":3,"viewNo":1,"txnRootHash":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}', '{"instId":1,"op":"PREPARE","ppTime":1503933947,"stateRootHash":null,"ppSeqNo":3,"viewNo":1,"txnRootHash":null,"digest":"953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6"}']} 2017-08-28 15:25:48,117 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,118 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933947, 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,121 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,129 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,129 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1503933947, 'op': 'PREPARE', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,129 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,142 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:48,159 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 3) from Node3:0 2017-08-28 15:25:48,163 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 3) at 250244.312540799 2017-08-28 15:25:48,164 | DEBUG | replica.py (2133) | send | Node1:0 sending Commit 2017-08-28 15:25:48,164 | TRACE | replica.py (2134) | send | Node1:0 sending COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,164 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:25:48,176 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,176 | DEBUG | replica.py ( 865) | processPrepare | Node1:0 received PREPARE(1, 3) from Node4:0 2017-08-28 15:25:48,176 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 0, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,179 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,180 | DEBUG | replica.py ( 865) | processPrepare | Node1:1 received PREPARE(1, 3) from Node4:1 2017-08-28 15:25:48,190 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '953659708c0673bb22b4a8a801c2f877ed6ff76a878428ccafb6654fa3127df6', 'viewNo': 1, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933947, 'ppSeqNo': 3} 2017-08-28 15:25:48,190 | DEBUG | replica.py ( 876) | processPrepare | Node1:1 processed incoming PREPARE (1, 3) 2017-08-28 15:25:48,209 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node2 2017-08-28 15:25:48,209 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node2 2017-08-28 15:25:48,210 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node3 2017-08-28 15:25:48,217 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node3 2017-08-28 15:25:48,218 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node4 2017-08-28 15:25:48,220 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node4 2017-08-28 15:25:48,230 | DEBUG | node.py (2566) | send | Node1 sending message COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:48,244 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:48,252 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}']} 2017-08-28 15:25:48,252 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:48,252 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}', '{"instId":1,"op":"COMMIT","viewNo":1,"ppSeqNo":3}']} 2017-08-28 15:25:48,253 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,263 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 1, 'instId': 0}, 'Node2') 2017-08-28 15:25:48,264 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,264 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,264 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 1, 'instId': 1}, 'Node2') 2017-08-28 15:25:48,265 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,289 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node2 2017-08-28 15:25:48,297 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node2 2017-08-28 15:25:48,298 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node3 2017-08-28 15:25:48,298 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node3 2017-08-28 15:25:48,299 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node4 2017-08-28 15:25:48,312 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"instId":0,"op":"COMMIT","viewNo":1,"ppSeqNo":3}' to Node4 2017-08-28 15:25:48,320 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 3) from Node2:0 2017-08-28 15:25:48,324 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} commits where f is 1 2017-08-28 15:25:48,331 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,331 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 3) from Node2:1 2017-08-28 15:25:48,333 | DEBUG | replica.py ( 929) | tryOrder | Node1:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} commits where f is 1 2017-08-28 15:25:48,333 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,334 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:48,354 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}']} 2017-08-28 15:25:48,354 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:48,354 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"instId":0,"viewNo":1,"op":"COMMIT"}', '{"ppSeqNo":3,"instId":1,"viewNo":1,"op":"COMMIT"}']} 2017-08-28 15:25:48,364 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,364 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 0, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,364 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,364 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,365 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'COMMIT', 'viewNo': 1, 'instId': 1, 'ppSeqNo': 3}, 'Node4') 2017-08-28 15:25:48,367 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,376 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:25:48,376 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:48,376 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":0}', '{"op":"COMMIT","viewNo":1,"ppSeqNo":3,"instId":1}']} 2017-08-28 15:25:48,376 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,384 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 1}, 'Node3') 2017-08-28 15:25:48,384 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} 2017-08-28 15:25:48,385 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,394 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 1}, 'Node3') 2017-08-28 15:25:48,394 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} 2017-08-28 15:25:48,422 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 3) from Node4:0 2017-08-28 15:25:48,427 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-08-28 15:25:48,428 | DEBUG | replica.py (1437) | doOrder | Node1:0 ordering COMMIT (1, 3) 2017-08-28 15:25:48,428 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:0 set last ordered as (1, 3) 2017-08-28 15:25:48,428 | DEBUG | replica.py (2133) | send | Node1:0 sending Ordered 2017-08-28 15:25:48,430 | TRACE | replica.py (2134) | send | Node1:0 sending ORDERED{'ppSeqNo': 3, 'ppTime': 1503933947, 'instId': 0, 'reqIdr': [], 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:25:48,440 | DEBUG | replica.py (1469) | order_3pc_key | Node1:0 ordered request (1, 3) 2017-08-28 15:25:48,441 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,443 | TRACE | message_processor.py ( 29) | discard | Node1:0 discarding message COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 0} because already ordered 3 phase message 2017-08-28 15:25:48,451 | DEBUG | replica.py ( 894) | processCommit | Node1:1 received COMMIT(1, 3) from Node4:1 2017-08-28 15:25:48,452 | TRACE | replica.py ( 925) | tryOrder | Node1:1 returning request to node 2017-08-28 15:25:48,452 | DEBUG | replica.py (1437) | doOrder | Node1:1 ordering COMMIT (1, 3) 2017-08-28 15:25:48,452 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (1, 3) 2017-08-28 15:25:48,467 | DEBUG | replica.py (2133) | send | Node1:1 sending Ordered 2017-08-28 15:25:48,472 | TRACE | replica.py (2134) | send | Node1:1 sending ORDERED{'ppSeqNo': 3, 'ppTime': 1503933947, 'instId': 1, 'reqIdr': [('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487)], 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 0, 'viewNo': 1} 2017-08-28 15:25:48,472 | DEBUG | replica.py (1469) | order_3pc_key | Node1:1 ordered request (1, 3) 2017-08-28 15:25:48,473 | DEBUG | replica.py ( 905) | processCommit | Node1:1 processed incoming COMMIT(1, 3) 2017-08-28 15:25:48,480 | TRACE | message_processor.py ( 29) | discard | Node1:1 discarding message COMMIT{'ppSeqNo': 3, 'viewNo': 1, 'instId': 1} because already ordered 3 phase message 2017-08-28 15:25:48,501 | DEBUG | node.py (1828) | processOrdered | Node1 executing Ordered batch 1 3 of 0 requests 2017-08-28 15:25:48,512 | DEBUG | ledger.py ( 52) | commitTxns | Committed 0 txns, 0 are uncommitted 2017-08-28 15:25:48,513 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:48,518 | TRACE | node.py (1839) | processOrdered | Node1 got ordered requests from backup replica 1 2017-08-28 15:25:48,565 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.737 seconds 2017-08-28 15:25:52,678 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:52,689 | DEBUG | zstack.py ( 661) | handlePingPong | Node1 got ping from b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,690 | DEBUG | zstack.py ( 643) | sendPingPong | Node1 ponged b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' 2017-08-28 15:25:52,697 | WARNING | batched.py ( 121) | flushOutBoxes | CONNECTION: Node1 rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' has been removed 2017-08-28 15:25:52,712 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message deque([b'po']) because CONNECTION: rid b'0@75wWXu$y-:}@dG*&HtG!H4kuYXW/&MPgUBV8%7' no longer available 2017-08-28 15:25:53,376 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 94 2017-08-28 15:25:53,377 | DEBUG | node.py (1980) | checkNodeRequestSpike | Node1 checking its request amount 2017-08-28 15:25:53,377 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-08-28 15:25:53,377 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 101 to run in 60 seconds 2017-08-28 15:25:53,464 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 14 2017-08-28 15:25:53,475 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-08-28 15:25:53,476 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 15 to run in 60 seconds 2017-08-28 15:25:53,668 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:53,669 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,683 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'reason': 25, 'op': 'INSTANCE_CHANGE', 'viewNo': 2}, 'Node4') 2017-08-28 15:25:53,683 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,690 | DEBUG | node.py (1896) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node4 2017-08-28 15:25:53,691 | DEBUG | node.py (1939) | do_view_change_if_possible | Node1 has no quorum for view 2 2017-08-28 15:25:53,691 | INFO | monitor.py ( 341) | isMasterThroughputTooLow | MONITORING: Node1 master throughput ratio 0.0 is lower than Delta 0.4. 2017-08-28 15:25:53,697 | INFO | node.py (1920) | processInstanceChange | VIEW CHANGE: Node1 found master degraded after receiving instance change message from Node4 2017-08-28 15:25:53,697 | DEBUG | throttler.py ( 31) | acquire | now: 250249.846015312, len(actionsLog): 0 2017-08-28 15:25:53,697 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-08-28 15:25:53,697 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-08-28 15:25:53,702 | INFO | node.py (2029) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 2 since Primary of master protocol instance degraded the performance 2017-08-28 15:25:53,707 | INFO | node.py (2032) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [249409.272495385, 249409.27303626] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 2.444435239012819} master request latencies: {} client avg request latencies: [{}, {'V4SGRU86Z58d6TV7PBUe6f': (2, 0.791428899494349), '6XiqM9DwYDgFnmyQQBFr72': (1, 0.8615774400241207)}] throughput: {0: 0, 1: 1.2272773490254318} master throughput: 0 total requests: 6 avg backup throughput: 1.2272773490254318 master throughput ratio: 0.0 2017-08-28 15:25:53,721 | DEBUG | node.py (2566) | send | Node1 sending message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:53,722 | DEBUG | node.py (1939) | do_view_change_if_possible | Node1 has no quorum for view 2 2017-08-28 15:25:53,735 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node2 2017-08-28 15:25:53,737 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node2 2017-08-28 15:25:53,738 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node3 2017-08-28 15:25:53,739 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node3 2017-08-28 15:25:53,747 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node4 2017-08-28 15:25:53,755 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"reason":25,"op":"INSTANCE_CHANGE","viewNo":2}' to Node4 2017-08-28 15:25:53,757 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:53,767 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,775 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'reason': 25, 'op': 'INSTANCE_CHANGE', 'viewNo': 2}, 'Node2') 2017-08-28 15:25:53,777 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,777 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,778 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'reason': 25, 'op': 'INSTANCE_CHANGE', 'viewNo': 2}, 'Node3') 2017-08-28 15:25:53,778 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} 2017-08-28 15:25:53,786 | DEBUG | node.py (1896) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node2 2017-08-28 15:25:53,786 | INFO | node.py (1935) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 2 from 1 2017-08-28 15:25:53,797 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action _check_view_change_completed with id 102 to run in 60 seconds 2017-08-28 15:25:53,797 | DEBUG | replica.py ( 453) | on_view_change_start | Node1:0 setting last prepared for master to (1, 3) 2017-08-28 15:25:53,798 | DEBUG | node.py (2127) | startViewChange | Node1 resetting monitor stats after view change 2017-08-28 15:25:53,798 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:25:53,798 | DEBUG | replica.py ( 406) | primaryName | Node1:0 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,799 | DEBUG | replica.py ( 406) | primaryName | Node1:1 setting primaryName for view no 2 to: None 2017-08-28 15:25:53,808 | INFO | node.py (2140) | startViewChange | VIEW CHANGE: Node1 changed to view 2, will start catchup now 2017-08-28 15:25:53,810 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,810 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:53,810 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:53,811 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:53,812 | DEBUG | node.py (2173) | start_catchup | Node1 reverted 0 batches before starting catch up 2017-08-28 15:25:53,820 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:53,820 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 0 2017-08-28 15:25:53,820 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:25:53,821 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 0 2017-08-28 15:25:53,826 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:53,826 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 0 2017-08-28 15:25:53,827 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:53,827 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 0 2017-08-28 15:25:53,838 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-08-28 15:25:53,839 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-08-28 15:25:53,839 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-08-28 15:25:53,848 | DEBUG | node.py (1927) | processInstanceChange | Node1 received instance change message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} but did not find the master to be slow or has already sent an instance change message 2017-08-28 15:25:53,850 | DEBUG | node.py (1896) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} from Node3 2017-08-28 15:25:53,857 | INFO | message_processor.py ( 29) | discard | Node1 discarding message INSTANCE_CHANGE{'reason': 25, 'viewNo': 2} because Received instance change request with view no 2 which is not more than its view no 2 2017-08-28 15:25:53,870 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:53,871 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,871 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:53,880 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:53,880 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:53,880 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,888 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:53,888 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:53,888 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:53,897 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:53,897 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:53,898 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:53,912 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:25:53,923 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:53,923 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:53,927 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:53,928 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,941 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:25:53,942 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,942 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,943 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:25:53,943 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,948 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:53,948 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:53,948 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:53,949 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,949 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:53,949 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,949 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,949 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:53,950 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,950 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:53,950 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:53,950 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:53,950 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,951 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:53,951 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,951 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,977 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:53,978 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:53,996 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:25:53,997 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,009 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,019 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,020 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,028 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,040 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,047 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,056 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:54,057 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:54,063 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,064 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,075 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:54,076 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:54,080 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,080 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-08-28 15:25:54,086 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' 2017-08-28 15:25:54,094 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:54,106 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:54,117 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:25:54,121 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:54,122 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}']} 2017-08-28 15:25:54,124 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,130 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:54,131 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,142 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,142 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-08-28 15:25:54,142 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,146 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:54,146 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:54,146 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', '{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']} 2017-08-28 15:25:54,146 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,146 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:25:54,146 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,147 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,147 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node2') 2017-08-28 15:25:54,147 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,173 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node3 2017-08-28 15:25:54,173 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,173 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,174 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node3 2017-08-28 15:25:54,187 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,187 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,193 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node2 2017-08-28 15:25:54,200 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,200 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,200 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 0 is latest 2017-08-28 15:25:54,200 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-08-28 15:25:54,209 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,209 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,209 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,209 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,210 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,210 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:25:54,217 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:25:54,217 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-08-28 15:25:54,218 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,218 | DEBUG | node.py ( 923) | checkInstances | Node1 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node3', 'Node4'} 2017-08-28 15:25:54,229 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:54,229 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-08-28 15:25:54,229 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,230 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-08-28 15:25:54,231 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,240 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-08-28 15:25:54,241 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,246 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 2 2017-08-28 15:25:54,246 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-08-28 15:25:54,247 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-08-28 15:25:54,253 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node2 2017-08-28 15:25:54,253 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,261 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,262 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,274 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,274 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:25:54,281 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:54,281 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,282 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,282 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:25:54,289 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:54,300 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:54,300 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,300 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:25:54,306 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:54,317 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:54,334 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,342 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:54,342 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":0},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,353 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,353 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:54,353 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,363 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,364 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node4') 2017-08-28 15:25:54,364 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 0, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-08-28 15:25:54,365 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node4 2017-08-28 15:25:54,378 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,379 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,385 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:25:54,386 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None} from Node4 2017-08-28 15:25:54,387 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,394 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 4 with 4 2017-08-28 15:25:54,395 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 0 is latest 2017-08-28 15:25:54,413 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:54,425 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:54,425 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:54,436 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:54,436 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,436 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,445 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,445 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,447 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,447 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,457 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,458 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:54,460 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,464 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,474 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:25:54,474 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,475 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,475 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:25:54,486 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,487 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,494 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,495 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,512 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,514 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,524 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,524 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:25:54,536 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:54,536 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,548 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-08-28 15:25:54,549 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' 2017-08-28 15:25:54,557 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:54,567 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-08-28 15:25:54,570 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-08-28 15:25:54,571 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-08-28 15:25:54,581 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-08-28 15:25:54,583 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:25:54,606 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:54,607 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:54,615 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:54,615 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,624 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,625 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,625 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,631 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,632 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,639 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:25:54,640 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:54,647 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-08-28 15:25:54,653 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,663 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,663 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,672 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,672 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-08-28 15:25:54,675 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,676 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,683 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:54,684 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-08-28 15:25:54,701 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,702 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:25:54,702 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,703 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,712 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node2') 2017-08-28 15:25:54,712 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,713 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,732 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,744 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:25:54,744 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,746 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,747 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node3 2017-08-28 15:25:54,756 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,767 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,768 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:25:54,768 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,774 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,774 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-08-28 15:25:54,779 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-08-28 15:25:54,779 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:25:54,780 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,780 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:54,791 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:54,792 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:54,792 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:25:54,803 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:25:54,809 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-08-28 15:25:54,809 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,809 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-08-28 15:25:54,819 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-08-28 15:25:54,820 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:54,820 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-08-28 15:25:54,827 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:25:54,831 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-08-28 15:25:54,832 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:54,832 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-08-28 15:25:54,844 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:54,844 | DEBUG | node.py ( 837) | _ask_for_ledger_status | Node1 asking Node4 for ledger status of ledger 1 2017-08-28 15:25:54,845 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 1 2017-08-28 15:25:54,849 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-08-28 15:25:54,849 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-08-28 15:25:54,857 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node2 2017-08-28 15:25:54,866 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,874 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:54,884 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:54,888 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,898 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:54,901 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:54,902 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:54,902 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,902 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:54,903 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:54,912 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 4 msgs to Node4 into one transmission 2017-08-28 15:25:54,913 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:54,924 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:54,930 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:54,942 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:54,952 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,963 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:54,964 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:54,976 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,976 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,976 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,982 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,983 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node4') 2017-08-28 15:25:54,993 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-08-28 15:25:54,994 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:25:55,006 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:55,006 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:55,007 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:25:55,013 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None} from Node4 2017-08-28 15:25:55,014 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:55,014 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-08-28 15:25:55,028 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-08-28 15:25:55,061 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:25:55,062 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,064 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:55,076 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,076 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,079 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:25:55,085 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,093 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,093 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:25:55,093 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,094 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:55,094 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:55,101 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}']} 2017-08-28 15:25:55,102 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,114 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,115 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,122 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,122 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,123 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,123 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:55,134 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:55,134 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}']} 2017-08-28 15:25:55,142 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,142 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,145 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,146 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,159 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,160 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,184 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:25:55,186 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-08-28 15:25:55,195 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,200 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-08-28 15:25:55,218 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:55,219 | DEBUG | node.py (2566) | send | Node1 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-08-28 15:25:55,229 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-08-28 15:25:55,240 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,249 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:55,256 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node2 2017-08-28 15:25:55,257 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-08-28 15:25:55,264 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,264 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:55,278 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node3 2017-08-28 15:25:55,285 | DEBUG | batched.py ( 99) | flushOutBoxes | Node1 batching 2 msgs to Node4 into one transmission 2017-08-28 15:25:55,286 | TRACE | batched.py ( 100) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-08-28 15:25:55,286 | TRACE | batched.py ( 108) | flushOutBoxes | Node1 sending payload to Node4: b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' 2017-08-28 15:25:55,299 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"signature":null,"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT\\",\\"ppSeqNo\\":6,\\"txnSeqNo\\":15,\\"viewNo\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH"}' to Node4 2017-08-28 15:25:55,302 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 3 messages through listener 2017-08-28 15:25:55,322 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,331 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH'}, 'Node2') 2017-08-28 15:25:55,339 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']} 2017-08-28 15:25:55,345 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,348 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:25:55,354 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,355 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,360 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node2') 2017-08-28 15:25:55,364 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,365 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:25:55,375 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}'], 'op': 'BATCH'}, 'Node3') 2017-08-28 15:25:55,378 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0}}']} 2017-08-28 15:25:55,379 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,388 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,389 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,389 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,390 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-08-28 15:25:55,398 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,404 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:55,405 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH'}, 'Node4') 2017-08-28 15:25:55,407 | DEBUG | node.py (1254) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT","ppSeqNo":6,"txnSeqNo":15,"viewNo":0},"op":"MESSAGE_RESPONSE"}']} 2017-08-28 15:25:55,416 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,417 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,425 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,426 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,426 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node4') 2017-08-28 15:25:55,426 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 15, 'ppSeqNo': 6, 'viewNo': 0, 'ledgerId': 1, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-08-28 15:25:55,429 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node2 2017-08-28 15:25:55,439 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,445 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,446 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node2 2017-08-28 15:25:55,454 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,462 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,474 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node3 2017-08-28 15:25:55,475 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,485 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,494 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-08-28 15:25:55,508 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} that it does not need catchup 2017-08-28 15:25:55,508 | DEBUG | node.py (1461) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-08-28 15:25:55,513 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,514 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-08-28 15:25:55,514 | DEBUG | replica.py (2199) | _remove_ordered_from_queue | Node1:1 going to remove 0 Ordered messages from outbox 2017-08-28 15:25:55,514 | DEBUG | node.py (1869) | force_process_ordered | Node1 processed 0 Ordered batches for instance 1 before starting catch up 2017-08-28 15:25:55,515 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:25:55,523 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:25:55,524 | INFO | node.py (1471) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-08-28 15:25:55,524 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,524 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-08-28 15:25:55,530 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,531 | DEBUG | node.py (2427) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-08-28 15:25:55,539 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-08-28 15:25:55,546 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 3 ViewChangeDone messages 2017-08-28 15:25:55,546 | DEBUG | node.py (1550) | caught_up_for_current_view | Node1 does not have view change quorum for view 2 2017-08-28 15:25:55,546 | DEBUG | node.py (1534) | is_catchup_needed | Node1 is not caught up for the current view 2 2017-08-28 15:25:55,547 | DEBUG | node.py (1591) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-08-28 15:25:55,553 | DEBUG | node.py (1539) | is_catchup_needed | Node1 ordered till last prepared certificate 2017-08-28 15:25:55,553 | INFO | node.py (1519) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-08-28 15:25:55,554 | DEBUG | primary_decider.py ( 131) | send | Node1's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,567 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 2 ViewChangeDone messages 2017-08-28 15:25:55,568 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,568 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node3 2017-08-28 15:25:55,571 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,580 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,581 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node4 2017-08-28 15:25:55,582 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,590 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,590 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:25:55,599 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 15, 'ppSeqNo': 6, 'merkleRoot': 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT', 'ledgerId': 1, 'viewNo': 0} from Node4 2017-08-28 15:25:55,606 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,606 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 15 with 15 2017-08-28 15:25:55,607 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-08-28 15:25:55,620 | DEBUG | node.py (2566) | send | Node1 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node3', 'viewNo': 2} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:55,627 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node2 2017-08-28 15:25:55,628 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node2 2017-08-28 15:25:55,628 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node3 2017-08-28 15:25:55,628 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node3 2017-08-28 15:25:55,639 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node4 2017-08-28 15:25:55,645 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,15,"EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","name":"Node3","viewNo":2}' to Node4 2017-08-28 15:25:55,651 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:55,664 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,665 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'viewNo': 2}, 'Node2') 2017-08-28 15:25:55,665 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,675 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,675 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'viewNo': 2}, 'Node3') 2017-08-28 15:25:55,682 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,683 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2}, 'Node2') 2017-08-28 15:25:55,691 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2}, 'Node3') 2017-08-28 15:25:55,695 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 100 2017-08-28 15:25:55,704 | TRACE | node.py (1959) | checkPerformance | Node1 checking its performance 2017-08-28 15:25:55,705 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 103 to run in 10 seconds 2017-08-28 15:25:55,706 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,707 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 1 ViewChangeDone messages 2017-08-28 15:25:55,716 | DEBUG | primary_selector.py ( 265) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-08-28 15:25:55,717 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,717 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (3 >= 3) 2017-08-28 15:25:55,718 | DEBUG | primary_selector.py ( 214) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node3 2017-08-28 15:25:55,729 | DEBUG | primary_selector.py ( 237) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node3 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-08-28 15:25:55,729 | DEBUG | primary_selector.py ( 280) | _startSelection | Node1 starting selection 2017-08-28 15:25:55,730 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node3:0 for instance 0 (view 2) 2017-08-28 15:25:55,738 | INFO | node.py ( 503) | start_participating | Node1 started participating 2017-08-28 15:25:55,738 | DEBUG | replica.py ( 406) | primaryName | Node1:0 setting primaryName for view no 2 to: Node3:0 2017-08-28 15:25:55,738 | DEBUG | replica.py (1678) | _gc | Node1:0 cleaning up till (1, 3) 2017-08-28 15:25:55,739 | DEBUG | replica.py (1693) | _gc | Node1:0 found 3 3-phase keys to clean 2017-08-28 15:25:55,739 | DEBUG | replica.py (1695) | _gc | Node1:0 found 3 request keys to clean 2017-08-28 15:25:55,743 | DEBUG | replica.py ( 311) | h | Node1:0 set watermarks as 0 300 2017-08-28 15:25:55,753 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,753 | DISPLAY | primary_selector.py ( 290) | _startSelection | PRIMARY SELECTION: Node1:1 selected primary Node4:1 for instance 1 (view 2) 2017-08-28 15:25:55,754 | DEBUG | replica.py ( 406) | primaryName | Node1:1 setting primaryName for view no 2 to: Node4:1 2017-08-28 15:25:55,760 | DEBUG | replica.py (1678) | _gc | Node1:1 cleaning up till (1, 3) 2017-08-28 15:25:55,760 | DEBUG | replica.py (1693) | _gc | Node1:1 found 3 3-phase keys to clean 2017-08-28 15:25:55,769 | DEBUG | replica.py (1695) | _gc | Node1:1 found 3 request keys to clean 2017-08-28 15:25:55,769 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933503177919) from previous checkpoints 2017-08-28 15:25:55,770 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('6XiqM9DwYDgFnmyQQBFr72', 1503933947440487) from previous checkpoints 2017-08-28 15:25:55,770 | DEBUG | replica.py (1716) | _gc | Node1:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1503933495609306) from previous checkpoints 2017-08-28 15:25:55,778 | DEBUG | replica.py ( 311) | h | Node1:1 set watermarks as 0 300 2017-08-28 15:25:55,778 | DEBUG | replica.py ( 512) | _setup_for_non_master | Node1:1 Setting last ordered for non-master as (1, 3) 2017-08-28 15:25:55,779 | DEBUG | replica.py ( 321) | last_ordered_3pc | Node1:1 set last ordered as (2, 0) 2017-08-28 15:25:55,781 | DISPLAY | primary_selector.py ( 314) | _startSelection | VIEW CHANGE: Node1:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-08-28 15:25:55,794 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:55,809 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,809 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'name': 'Node3', 'viewNo': 2}, 'Node4') 2017-08-28 15:25:55,809 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,810 | DEBUG | node.py (1187) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2}, 'Node4') 2017-08-28 15:25:55,830 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} 2017-08-28 15:25:55,846 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 15, 'EnsWAN36gj3N5ye36x1B4f5W1jpritjXhxyAsDJgf1FT'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node3', 'viewNo': 2} because it already decided primary which is Node3:0 2017-08-28 15:25:55,872 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 2.205 seconds 2017-08-28 15:25:57,420 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1C got 1 messages through listener 2017-08-28 15:25:57,440 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on request 1503933957374783 2017-08-28 15:25:57,440 | TRACE | node.py (1370) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,441 | DEBUG | node.py (1416) | processClientInBox | Node1C processing b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' request SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,452 | DEBUG | node.py (1690) | processRequest | Node1 received client request: SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} from b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,462 | TRACE | propagator.py ( 164) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,469 | DEBUG | propagator.py ( 148) | propagate | Node1 propagating Rhx2qwaeiGyhU9vn4ynHSS request 1503933957374783 from client b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' 2017-08-28 15:25:57,469 | DEBUG | node.py (2566) | send | Node1 sending message PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:57,470 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since not finalised 2017-08-28 15:25:57,480 | TRACE | zstack.py ( 752) | transmitThroughListener | Node1C transmitting b'{"op":"REQACK","identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}' to b'9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR' through listener socket 2017-08-28 15:25:57,493 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node2 2017-08-28 15:25:57,494 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node2 2017-08-28 15:25:57,503 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node3 2017-08-28 15:25:57,504 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node3 2017-08-28 15:25:57,509 | DEBUG | zstack.py ( 721) | transmit | Node1 transmitting message b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node4 2017-08-28 15:25:57,522 | TRACE | batched.py ( 95) | flushOutBoxes | Node1 sending msg b'{"op":"PROPAGATE","senderClient":"9C6z!fc&gD7!wTtPlKQeS.=AY[vp\\/BMSXqDx@uRR","request":{"signature":"4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R","operation":{"type":"0","dest":"4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc","data":{"node_ip":"10.0.0.105","client_ip":"10.0.0.105","services":["VALIDATOR"],"node_port":9701,"client_port":9702,"alias":"Node5"}},"identifier":"Rhx2qwaeiGyhU9vn4ynHSS","reqId":1503933957374783}}' to Node4 2017-08-28 15:25:57,530 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 2 messages through listener 2017-08-28 15:25:57,547 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,555 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,556 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}}, 'Node4') 2017-08-28 15:25:57,558 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,565 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,575 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,584 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}}, 'Node3') 2017-08-28 15:25:57,589 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,599 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,604 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,611 | DEBUG | propagator.py ( 213) | forward | Node1 forwarding request ('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783) to 0 replicas 2017-08-28 15:25:57,612 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'reqId': 1503933957374783}} 2017-08-28 15:25:57,620 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,620 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since already forwarded 2017-08-28 15:25:57,634 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:57,661 | DEBUG | node.py (2217) | verifySignature | Node1 authenticated Rhx2qwaeiGyhU9vn4ynHSS signature on propagate request 1503933957374783 2017-08-28 15:25:57,662 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,662 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'op': 'PROPAGATE', 'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}}, 'Node2') 2017-08-28 15:25:57,672 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,673 | DEBUG | node.py (1738) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': '9C6z!fc&gD7!wTtPlKQeS.=AY[vp/BMSXqDx@uRR', 'request': {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783}} 2017-08-28 15:25:57,686 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} 2017-08-28 15:25:57,687 | DEBUG | propagator.py ( 245) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '4haQCmu59ZcvoUhKAXdvycKKpn4Jzh4Qp3KtehJzaPLeSxx5af1f3ZwawghbvKdWBVmzpsUXuqut9fjLUX9K8k4R', 'operation': {'type': '0', 'dest': '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', 'data': {'node_ip': '10.0.0.105', 'client_ip': '10.0.0.105', 'services': ['VALIDATOR'], 'node_port': 9701, 'client_port': 9702, 'alias': 'Node5'}}, 'identifier': 'Rhx2qwaeiGyhU9vn4ynHSS', 'reqId': 1503933957374783} to its replicas since already forwarded 2017-08-28 15:25:57,735 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.334 seconds 2017-08-28 15:25:57,771 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:57,782 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node4: PREPREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'viewNo': 2, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:25:57,782 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 1, 'ppTime': 1503933957, 'op': 'PREPREPARE', 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'instId': 1, 'discarded': 1, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'stateRootHash': None, 'txnRootHash': None, 'ledgerId': 0, 'viewNo': 2}, 'Node4') 2017-08-28 15:25:57,791 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'viewNo': 2, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:25:57,808 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:1 received PRE-PREPARE(2, 1) from Node4:1 at 250253.957281041 2017-08-28 15:25:57,816 | DEBUG | replica.py ( 934) | doPrepare | Node1:1 Sending PREPARE(2, 1) at 250253.964771261 2017-08-28 15:25:57,816 | DEBUG | replica.py (2133) | send | Node1:1 sending Prepare 2017-08-28 15:25:57,818 | TRACE | replica.py (2134) | send | Node1:1 sending PREPARE{'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'viewNo': 2, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933957, 'ppSeqNo': 1} 2017-08-28 15:25:57,820 | DEBUG | replica.py ( 917) | tryCommit | Node1:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'viewNo': 2, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933957, 'ppSeqNo': 1} 2017-08-28 15:25:57,820 | DEBUG | replica.py ( 612) | trackBatches | Node1:1 tracking batch for PREPREPARE{'ppTime': 1503933957, 'stateRootHash': None, 'viewNo': 2, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)], 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'discarded': 1, 'txnRootHash': None, 'instId': 1, 'ppSeqNo': 1, 'ledgerId': 0} with state root None 2017-08-28 15:25:57,820 | DEBUG | replica.py ( 840) | processPrePrepare | Node1:1 processed incoming PRE-PREPARE(2, 1) 2017-08-28 15:25:57,821 | TRACE | zstack.py ( 477) | _receiveFromListener | Node1 got 1 messages through listener 2017-08-28 15:25:57,822 | DEBUG | node.py (1239) | validateNodeMsg | Node1 received node message from Node3: PREPREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 2, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:25:57,822 | DEBUG | node.py (1201) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'op': 'PREPREPARE', 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 2, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'ppTime': 1503933957, 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': 1}, 'Node3') 2017-08-28 15:25:57,822 | DEBUG | node.py (1268) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 2, 'reqIdr': [['Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783]], 'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'discarded': 0, 'txnRootHash': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'instId': 0, 'ppSeqNo': 1, 'ledgerId': 0} 2017-08-28 15:25:57,857 | DEBUG | node.py (2566) | send | Node1 sending message PREPARE{'digest': '8b001054bf64371a96dd73d3a067374168bcbc3c75ed1843bd4fe65ffb320166', 'viewNo': 2, 'instId': 1, 'stateRootHash': None, 'txnRootHash': None, 'ppTime': 1503933957, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node3', 'Node4'] 2017-08-28 15:25:57,868 | DEBUG | replica.py ( 821) | processPrePrepare | Node1:0 received PRE-PREPARE(2, 1) from Node3:0 at 250254.01742318 2017-08-28 15:25:57,869 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'ppTime': 1503933957, 'stateRootHash': 'HSLBcVDRU2EzjcfCJCEzhiYPRHh3mtGuQgyfojQPPV5m', 'viewNo': 2, 'reqIdr': [('Rhx2qwaeiGyhU9vn4ynHSS', 1503933957374783)], 'digest': '8b001054bf